<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 27, 2021 at 11:50 PM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div>  Mark,<div><br></div><div>   </div><div><br></div><div>    Where did you run the little test program I sent you </div><div><br></div><div>1) when it produced </div><div><br></div><div>   The 1120 and negative number and   (was this on the compile server or on a compute node?)</div></div></blockquote><div><br></div><div>This is fine now. look at my last email. I was not using srun.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><div>2) when it produced the correct answer? (compile server or compute node?)</div><div><br></div><div>Do you run configure on a compile server (that has no GPUs) or a compute server that has GPUs</div></div></blockquote><div><br></div><div>You have to do everything on the compute nodes on Cori/gpu.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><div> Don't spend your time bisecting PETSc we know exactly where the problem is, we just don't see how it happens. </div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><div>   cuda.py, if it cannot find deviceQuery and if you did not provide a generation arch with -with-cuda-gencodearch=70, </div></div></blockquote><div><br></div><div>I thought I was not supposed to use that anymore. It sounds like it is optional. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div>runs a version of the little code I sent you to get the number but it is ??apparently?? producing garbage or not running on the compiler server and gives the wrong number 1120. </div></div></blockquote><div><br></div><div>Does PETSc use MPIEXEC to run this?</div><div><br></div><div>Note, I have not been able to get 'make check' to work on Cori/gpu. I use '-with-mpiexec=srun -G1 [-c 20]' and it fails to execute the tests.</div><div><br></div><div>OK, putting -with-cuda-gencodearch=70 back in has fixed this problem. It is running now.</div><div><br></div><div>Thanks,</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><div>   Just use the option -with-cuda-gencodearch=70  (you do not need to pass this information to any flags any more, just with this option and it will use it). </div><div><br></div><div>  Barry</div><div><br></div><div>Ideally we want it to figure it out automatically and this little test program in configure is suppose to do this but since that is not always working yet you should just use -with-cuda-gencodearch=70</div><div><br></div><div><br><div><br><blockquote type="cite"><div>On May 27, 2021, at 5:45 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr">FYI, I was running the test incorrectly:<div>03:38 cgpu12  ~/petsc_install$ srun -n 1 -G 1 ./a.out <br>70<br>70<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 26, 2021 at 10:21 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I had git bisect working and was 4 steps away when I got a new crash.<div>configure.log is empty.<br><div><br><div>19:15 1 cgpu02 (a531cba26b...)|BISECTING ~/petsc$ git bisect bad<br>Bisecting: 19 revisions left to test after this (roughly 4 steps)<br>[149e269f455574fbe8ce3ebaf42121ae7fdf0635] Merge branch 'tisaac/feature-spqr' into 'main'<br>19:16 cgpu02 (149e269f45...)|BISECTING ~/petsc$ ../arch-cori-gpu-opt-gcc.py PETSC_DIR=$PWD<br>===============================================================================<br>             Configuring PETSc to compile on your system                       <br>===============================================================================<br>*******************************************************************************<br>        CONFIGURATION CRASH  (Please send configure.log to <a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a>)<br>*******************************************************************************<br><br>EOL while scanning string literal (cuda.py, line 176)<br>  File "/global/u2/m/madams/petsc/config/configure.py", line 455, in petsc_configure<br>    framework = config.framework.Framework(['--configModules=PETSc.Configure','--optionsModule=config.compilerOptions']+sys.argv[1:], loadArgDB = 0)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 107, in __init__<br>    self.createChildren()<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 344, in createChildren<br>    self.getChild(moduleName)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 329, in getChild<br>    config.setupDependencies(self)<br>  File "/global/u2/m/madams/petsc/config/PETSc/Configure.py", line 80, in setupDependencies<br>    self.blasLapack    = framework.require('config.packages.BlasLapack',self)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 349, in require<br>    config = self.getChild(moduleName, keywordArgs)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 329, in getChild<br>    config.setupDependencies(self)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/packages/BlasLapack.py", line 21, in setupDependencies<br>    config.package.Package.setupDependencies(self, framework)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/package.py", line 151, in setupDependencies<br>    self.mpi         = framework.require('config.packages.MPI',self)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 349, in require<br>    config = self.getChild(moduleName, keywordArgs)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 329, in getChild<br>    config.setupDependencies(self)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/packages/MPI.py", line 73, in setupDependencies<br>    self.mpich   = framework.require('config.packages.MPICH', self)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 349, in require<br>    config = self.getChild(moduleName, keywordArgs)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 329, in getChild<br>    config.setupDependencies(self)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/packages/MPICH.py", line 16, in setupDependencies<br>    self.cuda            = framework.require('config.packages.cuda',self)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 349, in require<br>    config = self.getChild(moduleName, keywordArgs)<br>  File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 302, in getChild<br>    type   = __import__(moduleName, globals(), locals(), ['Configure']).Configure<br>19:16 cgpu02 (149e269f45...)|BISECTING ~/petsc$ ../arch-cori-gpu-opt-gcc.py PETSC_DIR=$PWD<br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 26, 2021 at 10:10 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 26, 2021 at 6:13 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="auto"><div><br></div><div>  What is HOST=cori09  Does it have GPUs?</div><div><br></div>  <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_164490976c8e07e028a8f1ce1f5cd42d6" target="_blank">https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_164490976c8e07e028a8f1ce1f5cd42d6</a><div><br></div><div>  Seems to clearly state</div><div><br></div><div><dl><dt style="background-color:rgb(239,239,240);border-top:1px solid rgb(216,220,222);border-bottom:1px solid rgb(238,243,245);padding:3px;font-family:"Trebuchet MS","DIN Pro",sans-serif;font-size:14px">int  <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp" title="" shape="rect" style="color:rgb(118,185,0)" target="_blank">cudaDeviceProp</a>::<a href="https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_164490976c8e07e028a8f1ce1f5cd42d6" shape="rect" style="color:rgb(118,185,0)" target="_blank">major</a> [inherited] </dt><dd style="margin-left:20px;margin-bottom:20px;font-family:"Trebuchet MS","DIN Pro",sans-serif;font-size:14px"><div style="margin-top:0.8em;margin-bottom:2em"><p style="margin-left:0px;margin-top:0px">Major compute capability </p><div><br></div><div><br></div><div>Mark, please compile and run this program on the machine you are running configure on</div><div><br></div><div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(157,32,111)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(105,103,149)">#include</span><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures"><stdio.h></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(105,103,149)"><span style="font-variant-ligatures:no-common-ligatures">#include</span><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(157,32,111)"><cuda.h></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(157,32,111)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(105,103,149)">#include</span><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures"><cuda_runtime.h></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(157,32,111)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(105,103,149)">#include</span><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures"><cuda_runtime_api.h></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(157,32,111)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(105,103,149)">#include</span><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures"><cuda_device_runtime_api.h></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(45,150,30)"><span style="font-variant-ligatures:no-common-ligatures">int</span><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(74,0,255)">main</span><span style="font-variant-ligatures:no-common-ligatures">(</span><span style="font-variant-ligatures:no-common-ligatures">int</span><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)">arg</span><span style="font-variant-ligatures:no-common-ligatures">,</span><span style="font-variant-ligatures:no-common-ligatures">char</span><span style="font-variant-ligatures:no-common-ligatures"> **</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)">args</span><span style="font-variant-ligatures:no-common-ligatures">)</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">{</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(45,150,30)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(194,0,255)">struct</span><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures">cudaDeviceProp</span><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)">dp</span><span style="font-variant-ligatures:no-common-ligatures">;</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">                cudaGetDeviceProperties(&dp, 0);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">                printf(</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(157,32,111)">"%d\n"</span><span style="font-variant-ligatures:no-common-ligatures">,10*dp.major+dp.minor);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;min-height:16px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">                </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(45,150,30)">int</span><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)">major</span><span style="font-variant-ligatures:no-common-ligatures">,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)">minor</span><span style="font-variant-ligatures:no-common-ligatures">;</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><span style="white-space:pre-wrap">          </span>cuDeviceGetAttribute(&major, CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR, 0);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">cuDeviceGetAttribute(&minor, CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR, 0);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">                printf(</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(157,32,111)">"%d\n"</span><span style="font-variant-ligatures:no-common-ligatures">,10*major+minor);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">                </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(194,0,255)">return</span><span style="font-variant-ligatures:no-common-ligatures">(0);</span></div></div></div></dd></dl></div></div></div></blockquote><div>Probably, you need to check the return code of these two function calls to make sure they are correct.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="auto"><div><dl><dd style="margin-left:20px;margin-bottom:20px;font-family:"Trebuchet MS","DIN Pro",sans-serif;font-size:14px"><div style="margin-top:0.8em;margin-bottom:2em"><div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">}</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;min-height:16px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;min-height:16px">This is what I get </div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;min-height:16px"><br></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;min-height:16px"><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">$ nvcc mytest.c -lcuda</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">~/petsc</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)"><b> (main=)</b></span><span style="font-variant-ligatures:no-common-ligatures"> arch-main</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">$ ./a.out</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">70</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">70</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;line-height:normal">Which is exactly what it is suppose to do.</div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div>Barry</div></div></div></dd></dl><div><br><blockquote type="cite"><div>On May 26, 2021, at 5:31 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:</div><br><div><div><br>   Yes, this code which I guess never got hit before <br><br>cudaDeviceProp dp;                cudaGetDeviceProperties(&dp, 0);                printf("%d\n",10*dp.major+dp.minor);                return(0);;<br><br>is using the wrong property for the generation. <br><br>  Back to the CUDA documentation for the correct information. <br><br><br><br><blockquote type="cite">On May 26, 2021, at 3:47 PM, Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank">jacob.fai@gmail.com</a>> wrote:<br><br>1120 sounds suspiciously like some CUDA version rather than architecture or compute capability…<br><br>Best regards,<br><br>Jacob Faibussowitsch<br>(Jacob Fai - booss - oh - vitch)<br>Cell: +1 (312) 694-3391<br><br><blockquote type="cite">On May 26, 2021, at 22:29, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br><br>I started to get this error today on Cori. <br><br>nvcc fatal   : Unsupported gpu architecture 'compute_1120'<br><br>I am pretty sure I had a clean build but I can redo it if you don't know where this is from.<br><br>Thanks,<br>Mark<br><configure.log><br></blockquote></blockquote><br></div></div></blockquote></div><br></div></div></div></blockquote></div></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div></div>