<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jan 22, 2021, at 2:11 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 22, 2021 at 12:57 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><br class=""><div class=""><div class=""><div style="font-family:"Trebuchet MS","DIN Pro",sans-serif;font-size:14px" class=""><p style="margin-top:0px" class="">The library is thread safe and its functions can be called from multiple host threads, even with the same <samp class="">handle</samp>. When multiple threads share the same handle, </p><p style="margin-top:0px" class=""><br class=""></p><p style="margin-top:0px" class="">extreme care needs to be taken when the handle configuration is changed because that change will affect potentially subsequent cuBLAS calls in all threads. </p><p style="margin-top:0px" class=""> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</p><div class=""><br class=""></div><p style="margin-top:0px" class="">It is even more true for the destruction of the handle. So it is not recommended that multiple thread share the same cuBLAS handle.</p><div class=""><br class=""></div><div style="font-size:18px" class="">From my reading of this there should be absolutely no issue. The handle configuration is never being changed. It should be set in PetscInitialize() and destroyed in PetscFinalize(). Since lazy configuration of cuBLAS is turned off (right?).</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">You had me add to init.c:</div><div class=""><br class=""></div><div class=""><div style="line-height:18px" class=""><div style="color:rgb(46,46,46);font-family:Menlo,Monaco,"Courier New",monospace;font-size:12px;white-space:pre" class=""><span style="font-weight:bold" class="">#if</span> defined(PETSC_HAVE_THREADSAFETY)</div><div style="color:rgb(46,46,46);font-family:Menlo,Monaco,"Courier New",monospace;font-size:12px;white-space:pre" class=""> ierr = PetscCUPMInitializeCheck();CHKERRQ(ierr);</div><div style="color:rgb(46,46,46);font-family:Menlo,Monaco,"Courier New",monospace;font-size:12px;white-space:pre" class=""><span style="font-weight:bold" class="">#endif</span></div><div style="color:rgb(46,46,46);font-family:Menlo,Monaco,"Courier New",monospace;font-size:12px;white-space:pre" class=""><span style="font-weight:bold" class=""><br class=""></span></div><div style="color:rgb(46,46,46);font-family:Menlo,Monaco,"Courier New",monospace;font-size:12px;white-space:pre" class=""><span style="font-weight:bold" class=""><br class=""></span></div>because getHandle, which does lazy initialization, was not thread safe. But I think it is now.</div><div style="line-height:18px" class=""><br class=""></div><div style="line-height:18px" class="">I am going to test without it and remove if it's OK (I'm sure it will be)<br class=""></div></div></div></div></div></blockquote><div><br class=""></div>No, you can't remove this because it initializes the device etc. You can't have all the threads trying to initialize the device.</div><div><br class=""></div><div> Barry</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""><div style="line-height:18px" class=""><div style="color:rgb(46,46,46);font-family:Menlo,Monaco,"Courier New",monospace;font-size:12px;white-space:pre" class=""><br class=""></div>And I don't know what they mean by "when the handle configuration is changed". It makes sense to me that the handle would not be thread safe. It is an object and it has state. The purpose of making an object, and not have one global object is to make it thread safe...</div></div><div style="line-height:18px" class=""><br class=""></div><div style="line-height:18px" class="">I can see that it is not thread safe. There were clear non-deterministic race conditions and it differed to a random degree from a serial run each time. A vector norm returned 0 sometimes when the first entry != 0.</div><div style="line-height:18px" class=""><br class=""></div><div style="line-height:18px" class="">Mark</div></div></div>
</div></blockquote></div><br class=""></body></html>