<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 21, 2020, at 2:12 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 20, 2020 at 10:28 PM Xiaoye S. Li <<a href="mailto:xsli@lbl.gov" class="">xsli@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div style="font-family: verdana, sans-serif; font-size: small;" class="">Mark,<br class=""></div><div style="font-family: verdana, sans-serif; font-size: small;" class="">thanks for debugging this! Indeed, I confirm -- that particular "free" should be regular free instead of cudaHostfree(), because that data structure is not allocated by cudaAllocHost(). I have been running this cuda code on Summit, somehow the bug didn't show up.</div></div></blockquote><div class=""><br class=""></div><div class="">Odd, but it seems to work fine for me now. eg, I get a speedup of 6x on a ~50K equation 3D systems (Q3 elements with 2 dof per vertex).</div><div class=""> </div></div></div></div></blockquote><div><br class=""></div><div>Mark, is it such speedup wrt the CPU version of SUPERLU_DIST? Or just the PETSc factorizations? </div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div style="font-family: verdana, sans-serif; font-size: small;" class=""><br class=""></div><div style="font-family: verdana, sans-serif; font-size: small;" class="">I just updated the master branch with this fix. Will be absorbed in a future release.</div><div style="font-family: verdana, sans-serif; font-size: small;" class=""><br class=""></div><div style="font-family: verdana, sans-serif; font-size: small;" class="">As for PRNTlevel>=2, perhaps check your cmake build script. It should be set to 0 for production build.</div><div style="font-family: verdana, sans-serif; font-size: small;" class=""><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">I don't see where that gets set. PRNTlevel does not seem to be in our repo. I see it in 'MAKE_INC/make.cuda_gpu: -DDEBUGlevel=0 -DPRNTlevel=1 -DPROFlevel=0', but I think it is set at >= 2. I have manually disabled the print statements (~ 5 places).</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Mark</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div style="font-family: verdana, sans-serif; font-size: small;" class=""></div><div style="font-family: verdana, sans-serif; font-size: small;" class="">Sherry</div><div style="font-family: verdana, sans-serif; font-size: small;" class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 19, 2020 at 6:32 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Also, we have PRNTlevel>=2 in SuperLU_dist. This is causing a lot of output. It's not clear where that is set (it's a #define)</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 19, 2020 at 9:28 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Sherry, I found the problem.<div class=""><br class=""></div><div class="">I added this print statement to dDestroy_LU</div><div class=""><br class=""></div><div class=""> nb = CEILING(nsupers, grid->npcol);<br class=""> for (i = 0; i < nb; ++i) <br class=""> if ( Llu->Lrowind_bc_ptr[i] ) {<br class=""><b class=""><font color="#ff0000" class=""> fprintf(stderr,"dDestroy_LU: GPU free Llu->Lnzval_bc_ptr[%d/%d] = %p, CPU free Llu->Lrowind_bc_ptr = %p\n",i,nb,Llu->Lnzval_bc_ptr[i],Llu->Lrowind_bc_ptr[i]);<br class=""></font></b> SUPERLU_FREE (Llu->Lrowind_bc_ptr[i]);<br class="">#ifdef GPU_ACC<br class=""> checkCuda(cudaFreeHost(Llu->Lnzval_bc_ptr[i]));<br class="">#else<br class=""> SUPERLU_FREE (Llu->Lnzval_bc_ptr[i]);<br class="">#endif<br class=""> }<br class=""></div><div class=""><br class=""></div><div class="">And I see:</div><div class=""><br class=""></div><div class=""> 1 SNES Function norm 1.245977692562e-04<br class=""><b class="">dDestroy_LU: GPU free Llu->Lnzval_bc_ptr[0/134] = 0x4ff9b000, CPU free Llu->Lrowind_bc_ptr = 0x4ff9a000<br class=""></b>ex112d: <a href="http://cudahook.cc:762" class="">cudahook.cc:762</a>: CUresult host_free_callback(void*): Assertion `cacheNode != __null' failed.<br class=""></div><div class=""><br class=""></div><div class="">THis looks like Lnzval_bc_ptr is on the CPU so I removed the GPU_ACC stuff and it works now.</div><div class=""><br class=""></div><div class="">I see this in distribution. Perhaps this a serial run bug?</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 19, 2020 at 5:58 PM Xiaoye S. Li <<a href="mailto:xsli@lbl.gov" target="_blank" class="">xsli@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div style="font-family: verdana, sans-serif; font-size: small;" class="">Mark,</div><div style="font-family: verdana, sans-serif; font-size: small;" class="">you should fork a branch of your own to do this.</div><div style="font-family: verdana, sans-serif; font-size: small;" class=""><br class=""></div><div style="font-family: verdana, sans-serif; font-size: small;" class="">Sherry</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 19, 2020 at 2:54 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">First, commit your changes to the superlu_dist branch, then rerun configure with<br class="">
<br class="">
—download-superlu_dist-commit=HEAD<br class="">
<br class="">
<br class="">
> On Apr 20, 2020, at 12:50 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class="">
> <br class="">
> I would like to modify SuperLU_dist but if I change the source and configure it says no need to reconfigure, use --force. I use --force and it seems to clobber my changes. Can I tell configure to use build but not download SuperLU?<br class="">
<br class="">
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div></div>
</div></blockquote></div><br class=""></body></html>