<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Sep 27, 2020, at 11:55 AM, Zhang, Hong <<a href="mailto:hongzhang@anl.gov" class="">hongzhang@anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Sep 25, 2020, at 8:09 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
Configure by default should find out the available GPU and build for that sm_* it should not require the user to set this (how the heck is the user going to know what to set?) If I remember correctly there is a utility available that gives this information.
<div class=""><br class="">
</div>
<div class=""> For generic builds like in package distributions I don't know how it should work, ideally all the possibilities would be available in the library and at run time the correct one will be utilized. </div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">For package distribution we should add as many possibilities as possible. To have maximum compatibility on CUDA 11, we can add</div></div></div></div></blockquote><div><br class=""></div><div> Hong</div><div><br class=""></div> This is good, I'll add this support. But what about compiling PETSc code that depends on a later feature? How to have it in the library but not break things when not supported?</div><div><br class=""></div><div> Barry</div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">
<div class=""><br class="">
</div>
<span class="">
<div class="">-gencode=arch=compute_52,code=sm_52 \ </div>
<div class="">-gencode=arch=compute_60,code=sm_60 \ </div>
<div class="">-gencode=arch=compute_61,code=sm_61 \ </div>
<div class="">-gencode=arch=compute_70,code=sm_70 \ </div>
<div class="">-gencode=arch=compute_75,code=sm_75</div>
<br class="">
The downside is longer compilation time and fatter binary.</span></div>
<div class=""><span class=""><br class="">
</span></div>
<div class=""><span class="">Hong (Mr.)<br class="">
</span><span class=""><br class="">
</span><span class=""><br class="">
</span><span class=""><br class="">
</span>
<blockquote type="cite" class="">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class=""> Barry</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Sep 25, 2020, at 5:49 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class=""> '--CUDAFLAGS=-arch=sm_70',<br class="">
<div class=""><br class="">
</div>
<div class="">seems to fix this.</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Sep 25, 2020 at 6:31 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">I see kokkos and hyper have a sm_70 flag, but I don't see one for PETSc.<br class="">
<br class="">
It looks like you have to specify this to get modern atomics to work in Cuda. I get:
<div class=""><br class="">
</div>
<div class="">/ccs/home/adams/petsc/include/petscaijdevice.h(99): error: no instance of overloaded function "atomicAdd" matches the argument list<br class="">
argument types are: (double *, double)<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I tried using a Kokkos configuration, thinking I could get these sm_70 flags, but that did not work.</div>
<div class=""><br class="">
</div>
<div class="">Any ideas?</div>
<div class=""><br class="">
</div>
<div class="">Mark</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div></blockquote></div><br class=""></body></html>