<div dir="ltr">That was a login node. Here is a compute node on SUMMIT:<div><br></div><div>12:45 130 adams/feature-mat-cuda *= ~/petsc/src/mat/tutorials$ nvidia-smi<br>Sat Sep 26 12:45:28 2020<br>+-----------------------------------------------------------------------------+<br>| NVIDIA-SMI 418.116.00 Driver Version: 418.116.00 CUDA Version: 10.1 |<br>|-------------------------------+----------------------+----------------------+<br>| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br>| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br>|===============================+======================+======================|<br>| 0 Tesla V100-SXM2... On | 00000004:04:00.0 Off | 0 |<br>| N/A 36C P0 36W / 300W | 0MiB / 16130MiB | 0% E. Process |<br>+-------------------------------+----------------------+----------------------+<br>| 1 Tesla V100-SXM2... On | 00000004:05:00.0 Off | 0 |<br>| N/A 41C P0 36W / 300W | 0MiB / 16130MiB | 0% E. Process |<br>+-------------------------------+----------------------+----------------------+<br>| 2 Tesla V100-SXM2... On | 00000035:03:00.0 Off | 0 |<br>| N/A 34C P0 36W / 300W | 0MiB / 16130MiB | 0% E. Process |<br>+-------------------------------+----------------------+----------------------+<br>| 3 Tesla V100-SXM2... On | 00000035:04:00.0 Off | 0 |<br>| N/A 42C P0 37W / 300W | 0MiB / 16130MiB | 0% E. Process |<br>+-------------------------------+----------------------+----------------------+<br><br>+-----------------------------------------------------------------------------+<br>| Processes: GPU Memory |<br>| GPU PID Type Process name Usage |<br>|=============================================================================|<br>| No running processes found |<br>+-----------------------------------------------------------------------------+<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 26, 2020 at 12:44 PM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">0:30 adams/feature-mat-cuda *= ~/petsc$ nvidia-smi<br>Sat Sep 26 12:42:37 2020<br>+-----------------------------------------------------------------------------+<br>| NVIDIA-SMI 418.116.00 Driver Version: 418.116.00 CUDA Version: 10.1 |<br>|-------------------------------+----------------------+----------------------+<br>| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br>| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br>|===============================+======================+======================|<br>| 0 Tesla V100-SXM2... On | 00000004:05:00.0 Off | 0 |<br>| N/A 38C P0 36W / 300W | 0MiB / 16130MiB | 0% E. Process |<br>+-------------------------------+----------------------+----------------------+<br>| 1 Tesla V100-SXM2... On | 00000035:04:00.0 Off | 0 |<br>| N/A 40C P0 38W / 300W | 0MiB / 16130MiB | 0% E. Process |<br>+-------------------------------+----------------------+----------------------+<br><br>+-----------------------------------------------------------------------------+<br>| Processes: GPU Memory |<br>| GPU PID Type Process name Usage |<br>|=============================================================================|<br>| No running processes found |<br>+-----------------------------------------------------------------------------+<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 25, 2020 at 9:36 PM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank">jacob.fai@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><blockquote type="cite"><div> Configure by default should find out the available GPU and build for that sm_* it should not require the user to set this (how the heck is the user going to know what to set?) If I remember correctly there is a utility available that gives this information. </div></blockquote>For CUDA I believe the tool is nvidia-smi. Should make sure this automatic detection works when configuring —with-batch though since login nodes might have different arch than compute.<div><br><div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div>Best regards,<br><br>Jacob Faibussowitsch<br>(Jacob Fai - booss - oh - vitch)<br>Cell: (312) 694-3391</div></div>
</div>
<div><br><blockquote type="cite"><div>On Sep 25, 2020, at 21:09, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:</div><br><div><div><div><br></div> Configure by default should find out the available GPU and build for that sm_* it should not require the user to set this (how the heck is the user going to know what to set?) If I remember correctly there is a utility available that gives this information. <div><br></div><div> For generic builds like in package distributions I don't know how it should work, ideally all the possibilities would be available in the library and at run time the correct one will be utilized. </div><div><br></div><div> Barry</div><div><br><div><br><blockquote type="cite"><div>On Sep 25, 2020, at 5:49 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr"> '--CUDAFLAGS=-arch=sm_70',<br><div><br></div><div>seems to fix this.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 25, 2020 at 6:31 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I see kokkos and hyper have a sm_70 flag, but I don't see one for PETSc.<br><br>It looks like you have to specify this to get modern atomics to work in Cuda. I get:<div><br></div><div>/ccs/home/adams/petsc/include/petscaijdevice.h(99): error: no instance of overloaded function "atomicAdd" matches the argument list<br> argument types are: (double *, double)<br></div><div><br></div><div>I tried using a Kokkos configuration, thinking I could get these sm_70 flags, but that did not work.</div><div><br></div><div>Any ideas?</div><div><br></div><div>Mark</div></div>
</blockquote></div>
</div></blockquote></div><br></div></div></div></blockquote></div><br></div></div></blockquote></div>
</blockquote></div>