<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jun 25, 2022, at 9:44 AM, Matthew Martineau <<a href="mailto:mmartineau@nvidia.com" class="">mmartineau@nvidia.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252" class="">
<div class="">
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto" class="">
Thanks - AmgX will accept unordered column indices.</div></div></div></blockquote><div><br class=""></div> Any performance hit? </div><div><br class=""></div><div> We can provide them sorted efficiently in the future but currently can only provide unordered column indices efficiently on the GPU.</div><div><br class=""></div><div> Barry<br class=""><blockquote type="cite" class=""><div class=""><div class="">
<div id="mail-editor-reference-message-container" dir="auto" class=""><br class="">
<hr style="display:inline-block;width:98%" tabindex="-1" class="">
<div id="divRplyFwdMsg" style="font-size: 11pt;" class=""><strong class="">From:</strong> Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>><br class="">
<strong class="">Sent:</strong> Saturday, 25 June 2022, 14:39<br class="">
<strong class="">To:</strong> Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>><br class="">
<strong class="">Cc:</strong> Matthew Martineau <<a href="mailto:mmartineau@nvidia.com" class="">mmartineau@nvidia.com</a>>; For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class="">
<strong class="">Subject:</strong> Re: [petsc-dev] MatMPIAIJGetLocalMat problem with GPUs<br class="">
</div>
<br class="">
<table bgcolor="#FFEB9C" border="1" class="">
<tbody class="">
<tr class="">
<td class=""><font face="verdana" size="1" class=""><b class="">External email: Use caution opening links or attachments</b>
</font></td>
</tr>
</tbody>
</table>
<br class="">
<div class="">
<div class=""><br class="">
</div>
Does AMGX require sorted column indices? (Python indentation notation below)
<div class=""><br class="">
</div>
<div class=""> If not </div>
<div class=""> just use <span class="" style="font-size:inherit">MatMPIAIJGetLocalMatMerge instead of </span>MatMPIAIJGetLocalMat.</div>
<div class=""><br class="">
</div>
<div class=""> If yes, </div>
<div class=""> on the first call</div>
<div class=""><br class="">
</div>
<div class=""> Mat tmplocal;</div>
<div class=""> PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_INITIAL_MATRIX, &tmplocal));</div>
<div class=""> PetscCall(MatConvert(tmplocal,MATSEQAIJCUSPARSE,&amgx->localA));</div>
<div class=""> PetscCall(MatDestroy(&tmplocal));</div>
<div class=""><br class="">
</div>
<div class=""> leave the later calls as is with</div>
<div class=""><br class="">
</div>
<div class=""> PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA);</div>
<div class=""><br class="">
</div>
<div class="">Eventually, someone will need to buckle down and write MatMPIAIJGetLocalMat_SeqAIJCUSPARSE, but that can be done later.</div>
<div class=""><br class="">
</div>
<div class="">Barry</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 25, 2022, at 9:13 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div dir="ltr" class=""><br class="">
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jun 24, 2022 at 1:54 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="" style=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 24, 2022, at 1:38 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">I am rearranging the code for clarity from the repo but I have:
<div class=""><br class="">
</div>
<div class=""> PetscBool is_dev_ptrs;<br class="">
PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_INITIAL_MATRIX, &amgx->localA));<br class="">
PetscCall(PetscObjectTypeCompareAny((PetscObject)amgx->localA, &is_dev_ptrs, MATAIJCUSPARSE, MATSEQAIJCUSPARSE, MATMPIAIJCUSPARSE, ""));<br class="">
PetscPrintf(PETSC_COMM_SELF,"<b class="">checking against mataijcusparse amgx->localA = %d</b>\n",is_dev_ptrs);<br class="">
PetscCall(PetscObjectTypeCompareAny((PetscObject)Pmat, &is_dev_ptrs, MATAIJCUSPARSE, MATSEQAIJCUSPARSE, MATMPIAIJCUSPARSE, ""));<br class="">
PetscPrintf(PETSC_COMM_SELF,"<b class="">checking against mataijcusparse Pmat = %d</b>\n",is_dev_ptrs);<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">And it seems to show that MatMPIAIJGetLocalMat takes a <font color="#ff0000" class="">
mpiaijcusparse</font> Mat and returns an <font color="#ff0000" class="">seqaij</font> mat (<font color="#ff0000" class="">see below</font>):</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Yes, this is how it currently behaves as Stefano has indicated. Thus it is not currently directly suitable for use with GPUs. As Stefano has indicated it has be revised to handle <span class="" style="color:rgb(255,0,0)">mpiaijcusparse </span>matrices correctly
in the same way that <span class="" style="font-size:inherit">MatMPIAIJGetLocalMatMerge has been revised for GPUs.</span></div>
<div class=""><br class="">
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">OK, sorry, I did not understand that this is not supported. We need a MatMPIAIJCusparseGetLocalMatMerge (I read this as supported with "hstack" format, unsorted?, by Stefano)</div>
<div class=""><br class="">
</div>
<div class="">What is the best way to proceed? Should we just convert to amgx->localA to mpiaijcusparse if Pmat is a cusparse matrix?<br class="">
</div>
<div class="">If so, should this code go in amgx or MatMPIAIJGetLocalMat(MAT_INITIAL_MATRIX) ?</div>
<div class="">Or should I add a MatMPIAIJCusparseGetLocalMatMerge that simply wraps these two calls for now?</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Mark</div>
<div class=""><br class="">
</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="" style="">
<div class=""></div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class=""><br class="">
</div>
<div class="">AMGX version 2.2.0.132-opensource<br class="">
Built on Jun 24 2022, 09:21:43<br class="">
Compiled with CUDA Runtime 11.5, using CUDA driver 11.5<br class="">
<b class="">checking against mataijcusparse amgx->localA = 0<br class="">
checking against mataijcusparse Pmat = 1</b><br class="">
l<font color="#ff0000" class="">ocalA_name seqaij<br class="">
Pmat_name mpiaijcusparse</font><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Matt's existing testing code (below) then <font color="#ff0000" class="">
shows</font> the types that conform with these tests and prints that I added.</div>
<div class=""><br class="">
</div>
<div class=""> // XXX DEBUG REMOVE<br class="">
const char* localA_name;<br class="">
PetscObjectGetType((PetscObject)amgx->localA, &localA_name);<br class="">
PetscPrintf(PETSC_COMM_SELF,"localA_name %s\n", localA_name);<br class="">
const char* Pmat_name;<br class="">
PetscObjectGetType((PetscObject)Pmat, &Pmat_name);<br class="">
PetscPrintf(PETSC_COMM_SELF,"Pmat_name %s\n", Pmat_name);<br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jun 24, 2022 at 10:00 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 24, 2022, at 8:58 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">And before we move to the MR, I think Matt found a clear problem:
<div class=""><br class="">
* PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA));<br class="">
returns "localA seqaij"</div>
<div class=""><br class="">
</div>
* And, oddly, PetscCall(MatSeqAIJGetArrayRead(amgx->localA, &amgx->values)); returns:
<div class="">"it seems to detect that the pointer is a device mapped pointer but that it is invalid"</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
It does not return a device mapped pointer, it returns a valid host pointer only. MatSeqAIJGetArrayRead() is intended to only return a host pointer, it cannot return a device pointer. MatSeqAIJCusparseGetArrayRead() returns device pointers and should be used
for this purpose.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class=""><br class="">
</div>
<div class="">Matt, lets just comment out the REUSE line and add another INITIAL line (destroying the old Mat of course), and lets press on.</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Looking at the code there is no way that simply using INITIAL instead of REUSE will make a code that does not work on the GPU run on the GPU. The <span class="" style="font-size:inherit">MatMPIAIJGetLocalMat() </span>returns only a MATSEQAIJ matrix regardless
of the INITIAL versus REUSE and one can never get a device pointer from a non-GPU matrix.</div>
<div class=""><br class="">
</div>
<div class=""> As noted by Stefano, the code either needs to use <span class="" style="font-size:inherit">MatMPIAIJGetLocalMatMerge</span><span class="" style="font-size:inherit"> () which does return a CUSPARSE matrix (but the columns are not supported)
or </span><span class="" style="font-size:inherit">MatMPIAIJGetLocalMat()</span></div>
<div class=""><span class="" style="font-size:inherit">needs to be updated to return a CUSPARSE matrix when the input MPI matrix is a CUSPARSE matrix.</span></div>
<div class=""><span class="" style="font-size:inherit"><br class="">
</span></div>
<div class=""><span class="" style="font-size:inherit"><br class="">
</span></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">We can keep the debugging code for now.</div>
<div class=""><br class="">
</div>
<div class="">We (PETSc) can work on this independently,</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Mark</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jun 24, 2022 at 8:51 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr" class="">I am not seeing this response, I see my "hstack" comment last.
<div class=""><a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fpetsc%2Fpetsc%2F-%2Fmerge_requests%2F4323&data=05%7C01%7Cmmartineau%40nvidia.com%7C416bbc844bf646be910a08da56b03085%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637917611977954097%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=m5I2%2F%2BpR5a4PKMZcs9la%2BQcGne6CZHe7vx0SDDUPNEU%3D&reserved=0" originalsrc="https://gitlab.com/petsc/petsc/-/merge_requests/4323" shash="vfnOBjig7YWq4M80gI3kTx62Z6B4tQzMViIS8ouaf9E9QPPmEt1qP35Z2a38R6yCwNLfLvwp4z9NdymL/TVI7802zXl9bLhqGp/cK79k4W9XnwcAPqeQXGYt+i3WsTcGcMX927wnPjqlYXl27qbaZvXk7AS3q4ZUvcLt+M5o/k4=" target="_blank" class="">https://gitlab.com/petsc/petsc/-/merge_requests/4323</a><br class="">
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jun 23, 2022 at 4:37 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="">
<div class=""><br class="">
</div>
I have responded in the MR, which has all the context and the code. Please move this conversation from petsc-dev to the MR. Note you can use the little cartoon cloud symbol (upper write of the sub window with my text) to reply to my post and keep everything
in a thread for clarity.
<div class=""><br class="">
</div>
<div class=""> We are confused because it seems you are trying a variety of things and we don't know how the different things you tried resulted in the multiple errors you reported.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 23, 2022, at 3:59 PM, Matthew Martineau <<a href="mailto:mmartineau@nvidia.com" target="_blank" class="">mmartineau@nvidia.com</a>> wrote:</div>
<br class="">
<div class="">
<div class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="">I checked in the changes and some debugging statements.<u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)"><u class=""></u> <u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:10pt; font-family:Consolas">PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA));</span><span class="" style="font-size:10pt; font-family:Consolas"><u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:10pt; font-family:Consolas">PetscCall(PetscObjectTypeCompareAny((PetscObject)amgx->localA, &is_dev_ptrs, MATAIJCUSPARSE, MATSEQAIJCUSPARSE, MATMPIAIJCUSPARSE, ""));<u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class=""><u class=""></u> <u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="">Then the call returns false. If we instead call PetscObjectTypeCompareAny on Pmat then it returns true. If you print the type of the matrices:<u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class=""><u class=""></u> <u class=""></u></span></div><p class="MsoNormal" style="margin:0cm 0cm 3pt; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:9pt; font-family:"Courier New"; color:rgb(29,28,29)">localA seqaij<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin:0cm 0cm 3pt; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:9pt; font-family:"Courier New"; color:rgb(29,28,29)">Pmat mpiaijcusparse<u class=""></u><u class=""></u></span></p>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class=""><u class=""></u> <u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="">If you subsequently call MatSeqAIJCUSPARSEGetArrayRead on localA then it errors (presumably because of the type mismatch).<u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class=""><u class=""></u> <u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="">If we call MatSeqAIJGetArrayRead on localA and then pass the `values` to AmgX it seems to detect that the pointer is a device mapped pointer but that it is invalid.<u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class=""><u class=""></u> <u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:10pt; font-family:Consolas">PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA));<u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:10pt; font-family:Consolas">PetscCall(MatSeqAIJGetArrayRead(amgx->localA, &amgx->values)); // Seems to return invalid pointer, but I’ll investigate more</span><span class="" style="font-size:10pt; font-family:Consolas"><u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class=""><u class=""></u> <u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="">This doesn’t reproduce if we call:<u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-family:Consolas"><u class=""></u> <u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:10pt; font-family:Consolas">PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_INITIAL_MATRIX, &amgx->localA));<u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-size:10pt; font-family:Consolas">PetscCall(MatSeqAIJGetArrayRead(amgx->localA, &amgx->values)); // Pointer appears to be valid and we converge<u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class=""><u class=""></u> <u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="">Essentially all I want to achieve is that when we are parallel, we fetch the local part of A and the device pointer to the matrix values from that structure so that we can pass to AmgX. Preferring whichever API calls are the most efficient.<span class=""> </span><u class=""></u><u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class=""><u class=""></u> <u class=""></u></span></div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class=""><u class=""></u> <u class=""></u></span></div>
<div class="">
<div class="" style="border-style:solid none none; border-top-width:1pt; border-top-color:rgb(225,225,225); padding:3pt 0cm 0cm">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<b class=""><span lang="EN-US" class="">From:</span></b><span lang="EN-US" class=""><span class=""> </span>Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="" style="color:blue; text-decoration:underline">stefano.zampini@gmail.com</a>><span class=""> </span><br class="">
<b class="">Sent:</b><span class=""> </span>23 June 2022 20:55<br class="">
<b class="">To:</b><span class=""> </span>Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="" style="color:blue; text-decoration:underline">mfadams@lbl.gov</a>><br class="">
<b class="">Cc:</b><span class=""> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="" style="color:blue; text-decoration:underline">bsmith@petsc.dev</a>>; For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank" class="" style="color:blue; text-decoration:underline">petsc-dev@mcs.anl.gov</a>>;
Matthew Martineau <<a href="mailto:mmartineau@nvidia.com" target="_blank" class="" style="color:blue; text-decoration:underline">mmartineau@nvidia.com</a>><br class="">
<b class="">Subject:</b><span class=""> </span>Re: [petsc-dev] MatMPIAIJGetLocalMat problem with GPUs<u class=""></u><u class=""></u></span></div>
</div>
</div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
<table border="1" cellpadding="0" class="" style="background-color:rgb(255,235,156)">
<tbody class="">
<tr class="">
<td class="" style="padding:0.75pt">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<b class=""><span class="" style="font-size:7.5pt; font-family:Verdana,sans-serif">External email: Use caution opening links or attachments</span></b><span class="" style="font-size:7.5pt; font-family:Verdana,sans-serif"></span><u class=""></u><u class=""></u></div>
</td>
</tr>
</tbody>
</table>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
<div class="">
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
The logic is wrong. It should check for MATSEQAIJCUSPARSE. <u class=""></u><u class=""></u></div>
</div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
<div class="">
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
On Thu, Jun 23, 2022, 21:36 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="" style="color:blue; text-decoration:underline">mfadams@lbl.gov</a>> wrote:<u class=""></u><u class=""></u></div>
</div>
<blockquote class="" style="border-style:none none none solid; border-left-width:1pt; border-left-color:rgb(204,204,204); padding:0cm 0cm 0cm 6pt; margin-left:4.8pt; margin-right:0cm">
<div class="">
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
<div class="">
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
On Thu, Jun 23, 2022 at 3:02 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="" style="color:blue; text-decoration:underline">bsmith@petsc.dev</a>> wrote:<u class=""></u><u class=""></u></div>
</div>
<blockquote class="" style="border-style:none none none solid; border-left-width:1pt; border-left-color:rgb(204,204,204); padding:0cm 0cm 0cm 6pt; margin-left:4.8pt; margin-right:0cm">
<div class="">
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
It looks like the current code copies the nonzero values to the CPU from the MPI matrix (with the calls PetscCall(MatSeqAIJGetArrayRead(mpimat->A,&aav));<br class="">
PetscCall(MatSeqAIJGetArrayRead(mpimat->B,&bav));, then copies them into the CPU memory of the Seq matrix. When the matrix entries are next accessed on the GPU it should automatically copy them down to the GPU. So the code looks ok even for GPUs. We'll need
to see the full error message with what the "invalid pointer" is.<u class=""></u><u class=""></u></div>
</div>
</blockquote>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
I showed Matt how to peek into offloadmask and he found that it is a host state, but this is not the issue. The access method should do the copy to the device.<u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
I am thinking the logic here might be wrong. (Matt fixed "VEC" --> "MAT" in the comparison below).<u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
Matt, is the issue that you are calling <b class="">MatSeqAIJCUSPARSEGetArrayRead</b><b class=""><span class="" style="font-family:"Courier New""> </span></b>and getting a host pointer?<u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
I think the state of amgx->localA after the call to MatSeqAIJCUSPARSEGetArrayRead should be "BOTH" because this copied the data to the device so they are both valid and you should have device data.<u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<span class="" style="font-family:"Courier New"">211 PetscBool is_dev_ptrs;<br class="">
212 PetscCall(PetscObjectTypeCompareAny((PetscObject)amgx->localA, &is_dev_ptrs, VECCUDA, VECMPICUDA, VECSEQCUDA, ""));<br class="">
213<br class="">
214 if (is_dev_ptrs) {<br class="">
<b class="">216 PetscCall(MatSeqAIJCUSPARSEGetArrayRead(amgx->localA, &amgx->values));<br class="">
</b>217 } else {<br class="">
219 PetscCall(MatSeqAIJGetArrayRead(amgx->localA, &amgx->values));<span class=""> </span><br class="">
220 }</span><u class=""></u><u class=""></u></div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<blockquote class="" style="border-style:none none none solid; border-left-width:1pt; border-left-color:rgb(204,204,204); padding:0cm 0cm 0cm 6pt; margin-left:4.8pt; margin-right:0cm">
<div class="">
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
Barry<u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
Yes this routine is terribly inefficient for GPU matrices, it needs to be specialized to not use the GPU memory but that is a separate issue from there being bugs in the current code.<u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
The code also seems to implicitly assume the parallel matrix has the same nonzero pattern with a reuse. This should be checked with each use by stashing the nonzero state of the matrix into the sequential matrix and making sure the parallel matrix has that
same stashed value each time. Currently if one changes the nonzero matrix of the parallel matrix one is likely to get random confusing crashes due to memory corruption. But likely not the problem here.<u class=""></u><u class=""></u></div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<br class="">
<br class="">
<u class=""></u><u class=""></u></div>
<blockquote class="" style="margin-top:5pt; margin-bottom:5pt">
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
On Jun 23, 2022, at 2:23 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="" style="color:blue; text-decoration:underline">mfadams@lbl.gov</a>> wrote:<u class=""></u><u class=""></u></div>
</div>
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
<div class="">
<div class="">
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
We have a bug in the AMGx test snes_tests-ex13_amgx in parallel.<u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
Matt Martineau found that MatMPIAIJGetLocalMat worked in the first pass in the code below, where the local matrix is created (INITIAL), but in the next pass, when "REUSE" is used, he sees an invalid pointer.<u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
Matt found that it does have offloadmask == CPU. <u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
Maybe it is missing logic to put the output in same state as the input?<u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
Any ideas on this or should I just dig into it?<u class=""></u><u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
<u class=""></u> <u class=""></u></div>
</div>
<div class="">
<div class="" style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">
Thanks,<u class=""></u><u class=""></u></div>
</div>
<div class="">
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New"; box-sizing:inherit; font-variant-ligatures:none; white-space:pre-wrap; border-radius:4px"><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">bool partial_setup_allowed = (pc->setupcalled && pc->flag != DIFFERENT_NONZERO_PATTERN);<u class=""></u><u class=""></u></span></pre>
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New""><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">199 if (amgx->nranks > 1) {<u class=""></u><u class=""></u></span></pre>
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New""><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">200 if (partial_setup_allowed) {<u class=""></u><u class=""></u></span></pre>
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New""><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">202 PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA)); // This path seems doesn't work by the time we reach AmgX API<u class=""></u><u class=""></u></span></pre>
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New""><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">203 } else {<u class=""></u><u class=""></u></span></pre>
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New""><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">205 PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_INITIAL_MATRIX, &amgx->localA)); // This path works<u class=""></u><u class=""></u></span></pre>
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New""><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">206 }<u class=""></u><u class=""></u></span></pre>
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New""><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">207 } else {<u class=""></u><u class=""></u></span></pre>
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New""><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">208 amgx->localA = Pmat;<u class=""></u><u class=""></u></span></pre>
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New""><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">209 }<u class=""></u><u class=""></u></span></pre>
<pre class="" style="margin:0cm 0cm 3pt; font-size:10pt; font-family:"Courier New""><span class="" style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">210</span></pre>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br class="">
</div>
</div>
</div></blockquote></div><br class=""></body></html>