<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Monaco;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Theoretically the access patterns can be worse, but our sparse operations output matrices with unordered columns, so the fine matrix being sorted shouldn’t impact the overall performance.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Barry Smith <bsmith@petsc.dev>
<br>
<b>Sent:</b> 25 June 2022 16:05<br>
<b>To:</b> Matthew Martineau <mmartineau@nvidia.com><br>
<b>Cc:</b> Mark Adams <mfadams@lbl.gov>; For users of the development version of PETSc <petsc-dev@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-dev] MatMPIAIJGetLocalMat problem with GPUs<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="1" cellpadding="0" style="background:#FFEB9C">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">
</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jun 25, 2022, at 9:44 AM, Matthew Martineau <<a href="mailto:mmartineau@nvidia.com">mmartineau@nvidia.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="color:#212121">Thanks - AmgX will accept unordered column indices.<o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> Any performance hit? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> We can provide them sorted efficiently in the future but currently can only provide unordered column indices efficiently on the GPU.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Barry<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div id="mail-editor-reference-message-container">
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><strong><span style="font-family:"Calibri",sans-serif">From:</span></strong> Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br>
<strong><span style="font-family:"Calibri",sans-serif">Sent:</span></strong> Saturday, 25 June 2022, 14:39<br>
<strong><span style="font-family:"Calibri",sans-serif">To:</span></strong> Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>><br>
<strong><span style="font-family:"Calibri",sans-serif">Cc:</span></strong> Matthew Martineau <<a href="mailto:mmartineau@nvidia.com">mmartineau@nvidia.com</a>>; For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a>><br>
<strong><span style="font-family:"Calibri",sans-serif">Subject:</span></strong> Re: [petsc-dev] MatMPIAIJGetLocalMat problem with GPUs<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="1" cellpadding="0" style="background:#FFEB9C">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">
</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> Does AMGX require sorted column indices? (Python indentation notation below)
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> If not <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> just use MatMPIAIJGetLocalMatMerge instead of MatMPIAIJGetLocalMat.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> If yes, <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> on the first call<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Mat tmplocal;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_INITIAL_MATRIX, &tmplocal));<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> PetscCall(MatConvert(tmplocal,MATSEQAIJCUSPARSE,&amgx->localA));<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> PetscCall(MatDestroy(&tmplocal));<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> leave the later calls as is with<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA);<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Eventually, someone will need to buckle down and write MatMPIAIJGetLocalMat_SeqAIJCUSPARSE, but that can be done later.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Barry<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jun 25, 2022, at 9:13 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Jun 24, 2022 at 1:54 PM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jun 24, 2022, at 1:38 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">I am rearranging the code for clarity from the repo but I have:
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> PetscBool is_dev_ptrs;<br>
PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_INITIAL_MATRIX, &amgx->localA));<br>
PetscCall(PetscObjectTypeCompareAny((PetscObject)amgx->localA, &is_dev_ptrs, MATAIJCUSPARSE, MATSEQAIJCUSPARSE, MATMPIAIJCUSPARSE, ""));<br>
PetscPrintf(PETSC_COMM_SELF,"<b>checking against mataijcusparse amgx->localA = %d</b>\n",is_dev_ptrs);<br>
PetscCall(PetscObjectTypeCompareAny((PetscObject)Pmat, &is_dev_ptrs, MATAIJCUSPARSE, MATSEQAIJCUSPARSE, MATMPIAIJCUSPARSE, ""));<br>
PetscPrintf(PETSC_COMM_SELF,"<b>checking against mataijcusparse Pmat = %d</b>\n",is_dev_ptrs);<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">And it seems to show that MatMPIAIJGetLocalMat takes a <span style="color:red">
mpiaijcusparse</span> Mat and returns an <span style="color:red">seqaij</span> mat (<span style="color:red">see below</span>):<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> Yes, this is how it currently behaves as Stefano has indicated. Thus it is not currently directly suitable for use with GPUs. As Stefano has indicated it has be revised to handle <span style="color:red">mpiaijcusparse </span>matrices correctly
in the same way that MatMPIAIJGetLocalMatMerge has been revised for GPUs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">OK, sorry, I did not understand that this is not supported. We need a MatMPIAIJCusparseGetLocalMatMerge (I read this as supported with "hstack" format, unsorted?, by Stefano)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What is the best way to proceed? Should we just convert to amgx->localA to mpiaijcusparse if Pmat is a cusparse matrix?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">If so, should this code go in amgx or MatMPIAIJGetLocalMat(MAT_INITIAL_MATRIX) ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Or should I add a MatMPIAIJCusparseGetLocalMatMerge that simply wraps these two calls for now?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mark<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">AMGX version 2.2.0.132-opensource<br>
Built on Jun 24 2022, 09:21:43<br>
Compiled with CUDA Runtime 11.5, using CUDA driver 11.5<br>
<b>checking against mataijcusparse amgx->localA = 0<br>
checking against mataijcusparse Pmat = 1</b><br>
l<span style="color:red">ocalA_name seqaij<br>
Pmat_name mpiaijcusparse</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Matt's existing testing code (below) then <span style="color:red">
shows</span> the types that conform with these tests and prints that I added.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> // XXX DEBUG REMOVE<br>
const char* localA_name;<br>
PetscObjectGetType((PetscObject)amgx->localA, &localA_name);<br>
PetscPrintf(PETSC_COMM_SELF,"localA_name %s\n", localA_name);<br>
const char* Pmat_name;<br>
PetscObjectGetType((PetscObject)Pmat, &Pmat_name);<br>
PetscPrintf(PETSC_COMM_SELF,"Pmat_name %s\n", Pmat_name);<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Jun 24, 2022 at 10:00 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jun 24, 2022, at 8:58 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">And before we move to the MR, I think Matt found a clear problem:
<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
* PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA));<br>
returns "localA seqaij"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">* And, oddly, PetscCall(MatSeqAIJGetArrayRead(amgx->localA, &amgx->values)); returns:
<o:p></o:p></p>
<div>
<p class="MsoNormal">"it seems to detect that the pointer is a device mapped pointer but that it is invalid"<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> It does not return a device mapped pointer, it returns a valid host pointer only. MatSeqAIJGetArrayRead() is intended to only return a host pointer, it cannot return a device pointer. MatSeqAIJCusparseGetArrayRead() returns device pointers
and should be used for this purpose.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Matt, lets just comment out the REUSE line and add another INITIAL line (destroying the old Mat of course), and lets press on.<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> Looking at the code there is no way that simply using INITIAL instead of REUSE will make a code that does not work on the GPU run on the GPU. The MatMPIAIJGetLocalMat() returns only a MATSEQAIJ matrix regardless of the INITIAL versus
REUSE and one can never get a device pointer from a non-GPU matrix.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> As noted by Stefano, the code either needs to use MatMPIAIJGetLocalMatMerge () which does return a CUSPARSE matrix (but the columns are not supported) or MatMPIAIJGetLocalMat()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">needs to be updated to return a CUSPARSE matrix when the input MPI matrix is a CUSPARSE matrix.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">We can keep the debugging code for now.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We (PETSc) can work on this independently,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mark<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Jun 24, 2022 at 8:51 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">I am not seeing this response, I see my "hstack" comment last.
<o:p></o:p></p>
<div>
<p class="MsoNormal"><a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fpetsc%2Fpetsc%2F-%2Fmerge_requests%2F4323&data=05%7C01%7Cmmartineau%40nvidia.com%7C21ebc26195dd454cc10508da56bc204a%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637917663254505838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=L5VcVZqJmQXngGpoWabR%2ByqfBCxOpJQy0x%2F5CIDCddU%3D&reserved=0" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/4323</a><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jun 23, 2022 at 4:37 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> I have responded in the MR, which has all the context and the code. Please move this conversation from petsc-dev to the MR. Note you can use the little cartoon cloud symbol (upper write of the sub window with my text) to reply to my post
and keep everything in a thread for clarity. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> We are confused because it seems you are trying a variety of things and we don't know how the different things you tried resulted in the multiple errors you reported.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jun 23, 2022, at 3:59 PM, Matthew Martineau <<a href="mailto:mmartineau@nvidia.com" target="_blank">mmartineau@nvidia.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">I checked in the changes and some debugging statements.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA));</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">PetscCall(PetscObjectTypeCompareAny((PetscObject)amgx->localA, &is_dev_ptrs, MATAIJCUSPARSE, MATSEQAIJCUSPARSE, MATMPIAIJCUSPARSE, ""));</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Then the call returns false. If we instead call PetscObjectTypeCompareAny on Pmat then it returns true. If you print the type of the matrices:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Courier New";color:#1D1C1D">localA seqaij</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Courier New";color:#1D1C1D">Pmat mpiaijcusparse</span><o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">If you subsequently call MatSeqAIJCUSPARSEGetArrayRead on localA then it errors (presumably because of the type mismatch).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">If we call MatSeqAIJGetArrayRead on localA and then pass the `values` to AmgX it seems to detect that the pointer is a device mapped pointer but that it is invalid.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA));</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">PetscCall(MatSeqAIJGetArrayRead(amgx->localA, &amgx->values)); // Seems to return invalid pointer, but I’ll investigate more</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">This doesn’t reproduce if we call:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_INITIAL_MATRIX, &amgx->localA));</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">PetscCall(MatSeqAIJGetArrayRead(amgx->localA, &amgx->values)); // Pointer appears to be valid and we converge</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Essentially all I want to achieve is that when we are parallel, we fetch the local part of A and the device pointer to the matrix values from that structure so that we can pass to AmgX. Preferring whichever API calls are the most efficient. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> <br>
<b>Sent:</b> 23 June 2022 20:55<br>
<b>To:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Cc:</b> Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>; For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>>; Matthew Martineau <<a href="mailto:mmartineau@nvidia.com" target="_blank">mmartineau@nvidia.com</a>><br>
<b>Subject:</b> Re: [petsc-dev] MatMPIAIJGetLocalMat problem with GPUs</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<table class="MsoNormalTable" border="1" cellpadding="0" style="background:#FFEB9C">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<div>
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif">External email: Use caution opening links or attachments</span></b><o:p></o:p></p>
</div>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">The logic is wrong. It should check for MATSEQAIJCUSPARSE. <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Jun 23, 2022, 21:36 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Jun 23, 2022 at 3:02 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> It looks like the current code copies the nonzero values to the CPU from the MPI matrix (with the calls PetscCall(MatSeqAIJGetArrayRead(mpimat->A,&aav));<br>
PetscCall(MatSeqAIJGetArrayRead(mpimat->B,&bav));, then copies them into the CPU memory of the Seq matrix. When the matrix entries are next accessed on the GPU it should automatically copy them down to the GPU. So the code looks ok even for GPUs. We'll need
to see the full error message with what the "invalid pointer" is.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I showed Matt how to peek into offloadmask and he found that it is a host state, but this is not the issue. The access method should do the copy to the device.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I am thinking the logic here might be wrong. (Matt fixed "VEC" --> "MAT" in the comparison below).<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Matt, is the issue that you are calling <b>MatSeqAIJCUSPARSEGetArrayRead</b><b><span style="font-family:"Courier New""> </span></b>and getting a host pointer?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">I think the state of amgx->localA after the call to MatSeqAIJCUSPARSEGetArrayRead should be "BOTH" because this copied the data to the device so they are both valid and you should have device data.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">211 PetscBool is_dev_ptrs;<br>
212 PetscCall(PetscObjectTypeCompareAny((PetscObject)amgx->localA, &is_dev_ptrs, VECCUDA, VECMPICUDA, VECSEQCUDA, ""));<br>
213<br>
214 if (is_dev_ptrs) {<br>
<b>216 PetscCall(MatSeqAIJCUSPARSEGetArrayRead(amgx->localA, &amgx->values));<br>
</b>217 } else {<br>
219 PetscCall(MatSeqAIJGetArrayRead(amgx->localA, &amgx->values)); <br>
220 }</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Yes this routine is terribly inefficient for GPU matrices, it needs to be specialized to not use the GPU memory but that is a separate issue from there being bugs in the current code.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> The code also seems to implicitly assume the parallel matrix has the same nonzero pattern with a reuse. This should be checked with each use by stashing the nonzero state of the matrix into the sequential matrix and making sure the parallel
matrix has that same stashed value each time. Currently if one changes the nonzero matrix of the parallel matrix one is likely to get random confusing crashes due to memory corruption. But likely not the problem here.<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">On Jun 23, 2022, at 2:23 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">We have a bug in the AMGx test snes_tests-ex13_amgx in parallel.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Matt Martineau found that MatMPIAIJGetLocalMat worked in the first pass in the code below, where the local matrix is created (INITIAL), but in the next pass, when "REUSE" is used, he sees an invalid pointer.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Matt found that it does have offloadmask == CPU. <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Maybe it is missing logic to put the output in same state as the input?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Any ideas on this or should I just dig into it?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
</div>
<div>
<pre style="margin-bottom:3.0pt;box-sizing:inherit;font-variant-ligatures:none;white-space:pre-wrap;border-radius:4px"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">bool partial_setup_allowed = (pc->setupcalled && pc->flag != DIFFERENT_NONZERO_PATTERN);</span><o:p></o:p></pre>
<pre style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">199 if (amgx->nranks > 1) {</span><o:p></o:p></pre>
<pre style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">200 if (partial_setup_allowed) {</span><o:p></o:p></pre>
<pre style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">202 PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA)); // This path seems doesn't work by the time we reach AmgX API</span><o:p></o:p></pre>
<pre style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">203 } else {</span><o:p></o:p></pre>
<pre style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">205 PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_INITIAL_MATRIX, &amgx->localA)); // This path works</span><o:p></o:p></pre>
<pre style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">206 }</span><o:p></o:p></pre>
<pre style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">207 } else {</span><o:p></o:p></pre>
<pre style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">208 amgx->localA = Pmat;</span><o:p></o:p></pre>
<pre style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">209 }</span><o:p></o:p></pre>
<pre style="margin-bottom:3.0pt"><span style="font-size:9.0pt;font-family:"Monaco",serif;color:#1D1C1D">210</span><o:p></o:p></pre>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>