<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Thanks. On machines such as this one where you have to use $MPIEXEC to run code you will still need to provide the generation with -with-cuda-gencodearch=70. On systems where it can directly query the GPU without MPIEXEC it will automatically produce the correct result. Otherwise it will guess by compiling for different generations but this can produce an incorrect answer. <div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 7:59 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 27, 2021 at 11:50 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div> Mark,<div class=""><br class=""></div><div class=""> </div><div class=""><br class=""></div><div class=""> Where did you run the little test program I sent you </div><div class=""><br class=""></div><div class="">1) when it produced </div><div class=""><br class=""></div><div class=""> The 1120 and negative number and (was this on the compile server or on a compute node?)</div></div></blockquote><div class=""><br class=""></div><div class="">This is fine now. look at my last email. I was not using srun.</div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div><div class="">2) when it produced the correct answer? (compile server or compute node?)</div><div class=""><br class=""></div><div class="">Do you run configure on a compile server (that has no GPUs) or a compute server that has GPUs</div></div></blockquote><div class=""><br class=""></div><div class="">You have to do everything on the compute nodes on Cori/gpu.</div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div><div class=""> Don't spend your time bisecting PETSc we know exactly where the problem is, we just don't see how it happens. </div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div><div class=""> cuda.py, if it cannot find deviceQuery and if you did not provide a generation arch with -with-cuda-gencodearch=70, </div></div></blockquote><div class=""><br class=""></div><div class="">I thought I was not supposed to use that anymore. It sounds like it is optional. </div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class="">runs a version of the little code I sent you to get the number but it is ??apparently?? producing garbage or not running on the compiler server and gives the wrong number 1120. </div></div></blockquote><div class=""><br class=""></div><div class="">Does PETSc use MPIEXEC to run this?</div><div class=""><br class=""></div><div class="">Note, I have not been able to get 'make check' to work on Cori/gpu. I use '-with-mpiexec=srun -G1 [-c 20]' and it fails to execute the tests.</div><div class=""><br class=""></div><div class="">OK, putting -with-cuda-gencodearch=70 back in has fixed this problem. It is running now.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div><div class=""> Just use the option -with-cuda-gencodearch=70 (you do not need to pass this information to any flags any more, just with this option and it will use it). </div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class="">Ideally we want it to figure it out automatically and this little test program in configure is suppose to do this but since that is not always working yet you should just use -with-cuda-gencodearch=70</div><div class=""><br class=""></div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 27, 2021, at 5:45 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">FYI, I was running the test incorrectly:<div class="">03:38 cgpu12 ~/petsc_install$ srun -n 1 -G 1 ./a.out <br class="">70<br class="">70<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 26, 2021 at 10:21 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">I had git bisect working and was 4 steps away when I got a new crash.<div class="">configure.log is empty.<br class=""><div class=""><br class=""><div class="">19:15 1 cgpu02 (a531cba26b...)|BISECTING ~/petsc$ git bisect bad<br class="">Bisecting: 19 revisions left to test after this (roughly 4 steps)<br class="">[149e269f455574fbe8ce3ebaf42121ae7fdf0635] Merge branch 'tisaac/feature-spqr' into 'main'<br class="">19:16 cgpu02 (149e269f45...)|BISECTING ~/petsc$ ../arch-cori-gpu-opt-gcc.py PETSC_DIR=$PWD<br class="">===============================================================================<br class=""> Configuring PETSc to compile on your system <br class="">===============================================================================<br class="">*******************************************************************************<br class=""> CONFIGURATION CRASH (Please send configure.log to <a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" class="">petsc-maint@mcs.anl.gov</a>)<br class="">*******************************************************************************<br class=""><br class="">EOL while scanning string literal (cuda.py, line 176)<br class=""> File "/global/u2/m/madams/petsc/config/configure.py", line 455, in petsc_configure<br class=""> framework = config.framework.Framework(['--configModules=PETSc.Configure','--optionsModule=config.compilerOptions']+sys.argv[1:], loadArgDB = 0)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 107, in __init__<br class=""> self.createChildren()<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 344, in createChildren<br class=""> self.getChild(moduleName)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 329, in getChild<br class=""> config.setupDependencies(self)<br class=""> File "/global/u2/m/madams/petsc/config/PETSc/Configure.py", line 80, in setupDependencies<br class=""> self.blasLapack = framework.require('config.packages.BlasLapack',self)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 349, in require<br class=""> config = self.getChild(moduleName, keywordArgs)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 329, in getChild<br class=""> config.setupDependencies(self)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/packages/BlasLapack.py", line 21, in setupDependencies<br class=""> config.package.Package.setupDependencies(self, framework)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/package.py", line 151, in setupDependencies<br class=""> self.mpi = framework.require('config.packages.MPI',self)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 349, in require<br class=""> config = self.getChild(moduleName, keywordArgs)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 329, in getChild<br class=""> config.setupDependencies(self)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/packages/MPI.py", line 73, in setupDependencies<br class=""> self.mpich = framework.require('config.packages.MPICH', self)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 349, in require<br class=""> config = self.getChild(moduleName, keywordArgs)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 329, in getChild<br class=""> config.setupDependencies(self)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/packages/MPICH.py", line 16, in setupDependencies<br class=""> self.cuda = framework.require('config.packages.cuda',self)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 349, in require<br class=""> config = self.getChild(moduleName, keywordArgs)<br class=""> File "/global/u2/m/madams/petsc/config/BuildSystem/config/framework.py", line 302, in getChild<br class=""> type = __import__(moduleName, globals(), locals(), ['Configure']).Configure<br class="">19:16 cgpu02 (149e269f45...)|BISECTING ~/petsc$ ../arch-cori-gpu-opt-gcc.py PETSC_DIR=$PWD<br class=""></div></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 26, 2021 at 10:10 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 26, 2021 at 6:13 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div dir="auto" class=""><div class=""><br class=""></div><div class=""> What is HOST=cori09 Does it have GPUs?</div><div class=""><br class=""></div> <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_164490976c8e07e028a8f1ce1f5cd42d6" target="_blank" class="">https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_164490976c8e07e028a8f1ce1f5cd42d6</a><div class=""><br class=""></div><div class=""> Seems to clearly state</div><div class=""><br class=""></div><div class=""><dl class=""><dt style="background-color:rgb(239,239,240);border-top:1px solid rgb(216,220,222);border-bottom:1px solid rgb(238,243,245);padding:3px;font-family:"Trebuchet MS","DIN Pro",sans-serif;font-size:14px" class="">int <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp" title="" shape="rect" style="color:rgb(118,185,0)" target="_blank" class="">cudaDeviceProp</a>::<a href="https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_164490976c8e07e028a8f1ce1f5cd42d6" shape="rect" style="color:rgb(118,185,0)" target="_blank" class="">major</a> [inherited] </dt><dd style="margin-left:20px;margin-bottom:20px;font-family:"Trebuchet MS","DIN Pro",sans-serif;font-size:14px" class=""><div style="margin-top:0.8em;margin-bottom:2em" class=""><p style="margin-left:0px;margin-top:0px" class="">Major compute capability </p><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Mark, please compile and run this program on the machine you are running configure on</div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(157,32,111)" class=""><span style="font-variant-ligatures:no-common-ligatures;color:rgb(105,103,149)" class="">#include</span><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures" class=""><stdio.h></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(105,103,149)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#include</span><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(157,32,111)" class=""><cuda.h></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(157,32,111)" class=""><span style="font-variant-ligatures:no-common-ligatures;color:rgb(105,103,149)" class="">#include</span><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures" class=""><cuda_runtime.h></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(157,32,111)" class=""><span style="font-variant-ligatures:no-common-ligatures;color:rgb(105,103,149)" class="">#include</span><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures" class=""><cuda_runtime_api.h></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(157,32,111)" class=""><span style="font-variant-ligatures:no-common-ligatures;color:rgb(105,103,149)" class="">#include</span><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures" class=""><cuda_device_runtime_api.h></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(45,150,30)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">int</span><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(74,0,255)" class="">main</span><span style="font-variant-ligatures:no-common-ligatures" class="">(</span><span style="font-variant-ligatures:no-common-ligatures" class="">int</span><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)" class="">arg</span><span style="font-variant-ligatures:no-common-ligatures" class="">,</span><span style="font-variant-ligatures:no-common-ligatures" class="">char</span><span style="font-variant-ligatures:no-common-ligatures" class=""> **</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)" class="">args</span><span style="font-variant-ligatures:no-common-ligatures" class="">)</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">{</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(45,150,30)" class=""><span style="font-variant-ligatures:no-common-ligatures;color:rgb(194,0,255)" class="">struct</span><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures" class="">cudaDeviceProp</span><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)" class="">dp</span><span style="font-variant-ligatures:no-common-ligatures" class="">;</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> cudaGetDeviceProperties(&dp, 0);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> printf(</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(157,32,111)" class="">"%d\n"</span><span style="font-variant-ligatures:no-common-ligatures" class="">,10*dp.major+dp.minor);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;min-height:16px" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""></span><br class=""></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(45,150,30)" class="">int</span><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)" class="">major</span><span style="font-variant-ligatures:no-common-ligatures" class="">,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)" class="">minor</span><span style="font-variant-ligatures:no-common-ligatures" class="">;</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><span style="white-space:pre-wrap" class=""> </span>cuDeviceGetAttribute(&major, CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR, 0);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cuDeviceGetAttribute(&minor, CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR, 0);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> printf(</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(157,32,111)" class="">"%d\n"</span><span style="font-variant-ligatures:no-common-ligatures" class="">,10*major+minor);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(194,0,255)" class="">return</span><span style="font-variant-ligatures:no-common-ligatures" class="">(0);</span></div></div></div></dd></dl></div></div></div></blockquote><div class="">Probably, you need to check the return code of these two function calls to make sure they are correct.</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div dir="auto" class=""><div class=""><dl class=""><dd style="margin-left:20px;margin-bottom:20px;font-family:"Trebuchet MS","DIN Pro",sans-serif;font-size:14px" class=""><div style="margin-top:0.8em;margin-bottom:2em" class=""><div class=""><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">}</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;min-height:16px" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""></span><br class=""></div></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;min-height:16px" class="">This is what I get </div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;min-height:16px" class=""><br class=""></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;min-height:16px" class=""><div style="margin:0px;font-stretch:normal;line-height:normal" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">$ nvcc mytest.c -lcuda</span></div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">~/petsc</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)" class=""><b class=""> (main=)</b></span><span style="font-variant-ligatures:no-common-ligatures" class=""> arch-main</span></div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">$ ./a.out</span></div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">70</span></div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">70</span></div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div style="margin:0px;font-stretch:normal;line-height:normal" class="">Which is exactly what it is suppose to do.</div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class="">Barry</div></div></div></dd></dl><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 26, 2021, at 5:31 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:</div><br class=""><div class=""><div class=""><br class=""> Yes, this code which I guess never got hit before <br class=""><br class="">cudaDeviceProp dp; cudaGetDeviceProperties(&dp, 0); printf("%d\n",10*dp.major+dp.minor); return(0);;<br class=""><br class="">is using the wrong property for the generation. <br class=""><br class=""> Back to the CUDA documentation for the correct information. <br class=""><br class=""><br class=""><br class=""><blockquote type="cite" class="">On May 26, 2021, at 3:47 PM, Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank" class="">jacob.fai@gmail.com</a>> wrote:<br class=""><br class="">1120 sounds suspiciously like some CUDA version rather than architecture or compute capability…<br class=""><br class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: +1 (312) 694-3391<br class=""><br class=""><blockquote type="cite" class="">On May 26, 2021, at 22:29, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""><br class="">I started to get this error today on Cori. <br class=""><br class="">nvcc fatal : Unsupported gpu architecture 'compute_1120'<br class=""><br class="">I am pretty sure I had a clean build but I can redo it if you don't know where this is from.<br class=""><br class="">Thanks,<br class="">Mark<br class=""><configure.log><br class=""></blockquote></blockquote><br class=""></div></div></blockquote></div><br class=""></div></div></div></blockquote></div></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></blockquote></div></div>
</div></blockquote></div><br class=""></div></body></html>