<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#000000">Mark,<br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#000000">thanks for debugging this! Indeed, I confirm -- that particular "free" should be regular free instead of cudaHostfree(), because that data structure is not allocated by cudaAllocHost(). I have been running this cuda code on Summit, somehow the bug didn't show up.</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#000000"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#000000">I just updated the master branch with this fix. Will be absorbed in a future release.</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#000000"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#000000">As for PRNTlevel>=2, perhaps check your cmake build script. It should be set to 0 for production build.</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#000000"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#000000">Sherry</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#000000"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 19, 2020 at 6:32 PM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Also, we have PRNTlevel>=2 in SuperLU_dist. This is causing a lot of output. It's not clear where that is set (it's a #define)</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 19, 2020 at 9:28 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sherry, I found the problem.<div><br></div><div>I added this print statement to dDestroy_LU</div><div><br></div><div> nb = CEILING(nsupers, grid->npcol);<br> for (i = 0; i < nb; ++i) <br> if ( Llu->Lrowind_bc_ptr[i] ) {<br><b><font color="#ff0000"> fprintf(stderr,"dDestroy_LU: GPU free Llu->Lnzval_bc_ptr[%d/%d] = %p, CPU free Llu->Lrowind_bc_ptr = %p\n",i,nb,Llu->Lnzval_bc_ptr[i],Llu->Lrowind_bc_ptr[i]);<br></font></b> SUPERLU_FREE (Llu->Lrowind_bc_ptr[i]);<br>#ifdef GPU_ACC<br> checkCuda(cudaFreeHost(Llu->Lnzval_bc_ptr[i]));<br>#else<br> SUPERLU_FREE (Llu->Lnzval_bc_ptr[i]);<br>#endif<br> }<br></div><div><br></div><div>And I see:</div><div><br></div><div> 1 SNES Function norm 1.245977692562e-04<br><b>dDestroy_LU: GPU free Llu->Lnzval_bc_ptr[0/134] = 0x4ff9b000, CPU free Llu->Lrowind_bc_ptr = 0x4ff9a000<br></b>ex112d: cudahook.cc:762: CUresult host_free_callback(void*): Assertion `cacheNode != __null' failed.<br></div><div><br></div><div>THis looks like Lnzval_bc_ptr is on the CPU so I removed the GPU_ACC stuff and it works now.</div><div><br></div><div>I see this in distribution. Perhaps this a serial run bug?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 19, 2020 at 5:58 PM Xiaoye S. Li <<a href="mailto:xsli@lbl.gov" target="_blank">xsli@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)">Mark,</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)">you should fork a branch of your own to do this.</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)">Sherry</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 19, 2020 at 2:54 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">First, commit your changes to the superlu_dist branch, then rerun configure with<br>
<br>
—download-superlu_dist-commit=HEAD<br>
<br>
<br>
> On Apr 20, 2020, at 12:50 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> <br>
> I would like to modify SuperLU_dist but if I change the source and configure it says no need to reconfigure, use --force. I use --force and it seems to clobber my changes. Can I tell configure to use build but not download SuperLU?<br>
<br>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>