<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin:0px"><span style="margin:0px;font-weight:normal;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif"><span style="margin:0px;font-size:14.6667px;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255);display:inline !important">Pierre,</span></span></div>
I pushed a fix in branch hzhang/fix-matmatmult_aij_dense/release. <a href="https://gitlab.com/petsc/petsc/-/merge_requests/3667" id="LPlnk">https://gitlab.com/petsc/petsc/-/merge_requests/3667</a>
<div class="_Entity _EType_OWALinkPreview _EId_OWALinkPreview _EReadonly_1">
<div id="LPBorder_GTaHR0cHM6Ly9naXRsYWIuY29tL3BldHNjL3BldHNjLy0vbWVyZ2VfcmVxdWVzdHMvMzY2Nw.." class="LPBorder144335" style="width: 100%; margin-top: 16px; margin-bottom: 16px; position: relative; max-width: 800px; min-width: 424px;">
<table id="LPContainer144335" role="presentation" style="padding: 12px 36px 12px 12px; width: 100%; border-width: 1px; border-style: solid; border-color: rgb(200, 200, 200); border-radius: 2px;">
<tbody>
<tr valign="top" style="border-spacing: 0px;">
<td>
<div id="LPImageContainer144335" style="position: relative; margin-right: 12px; height: 160px; overflow: hidden;">
<a target="_blank" id="LPImageAnchor144335" href="https://gitlab.com/petsc/petsc/-/merge_requests/3667"><img id="LPThumbnailImageId144335" alt="" height="160" style="display: block;" width="160" src="https://assets.gitlab-static.net/uploads/-/system/project/avatar/13882401/PETSc_RBG-logo.png"></a></div>
</td>
<td style="width: 100%;">
<div id="LPTitle144335" style="font-size: 21px; font-weight: 300; margin-right: 8px; font-family: wf_segoe-ui_light, "Segoe UI Light", "Segoe WP Light", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12px;">
<a target="_blank" id="LPUrlAnchor144335" href="https://gitlab.com/petsc/petsc/-/merge_requests/3667" style="text-decoration: none; color: var(--themePrimary);">bugfix for MatMatMultSymbolic_MPIAIJ_MPIDense() when Bbn1 = 0. (!3667) · Merge Requests · PETSc
/ petsc</a></div>
<div id="LPDescription144335" style="font-size: 14px; max-height: 100px; color: rgb(102, 102, 102); font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12px; margin-right: 8px; overflow: hidden;">
Reported-by: Pierre Jolivet pierre@joliv.et Bb (column block size) cannot be zero; it leads to infinite loop in MatMatMultNumeric_MPIAIJ_MPIDense() with n=0</div>
<div id="LPMetadata144335" style="font-size: 14px; font-weight: 400; color: rgb(166, 166, 166); font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif;">
gitlab.com</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
Give it a try. Let me know if the bug is not fixed.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Your code is very helpful in debugging.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hong</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Pierre Jolivet <pierre@joliv.et><br>
<b>Sent:</b> Monday, March 1, 2021 12:51 AM<br>
<b>To:</b> Zhang, Hong <hzhang@mcs.anl.gov><br>
<b>Cc:</b> For users of the development version of PETSc <petsc-dev@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-dev] Infinite loop in A*B</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space"><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 1 Mar 2021, at 6:29 AM, Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="color:rgb(32,31,30); font-size:14.6667px; background-color:rgb(255,255,255); display:inline!important">Pierre,</span><br class="">
</div>
<div class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font color="#201f1e" class=""><span class="" style="font-size:14.6667px">This is a bug in MatMatMultSymbolic_MPIAIJ_MPIDense() during optimization of block column size of B. Run your code with</span></font></div>
<div class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font color="#201f1e" class=""><span class="" style="font-size:14.6667px">'-matmatmult_Bbn 1', the infinite loop should not occur.</span></font></div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Thanks Hong, I can confirm this option makes the more complex use case run smoothly as well. </div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font color="#201f1e" class=""><span class="" style="font-size:14.6667px">I'll try to figure out a fix tomorrow.<br class="">
</span></font></div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Great.</div>
<div><br class="">
</div>
<div>Thanks,</div>
<div>Pierre</div>
<div><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font color="#201f1e" class=""><span class="" style="font-size:14.6667px">Hong</span></font></div>
<div id="x_appendonsend" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
</div>
<hr tabindex="-1" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; display:inline-block; width:592.890625px">
<span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="x_divRplyFwdMsg" dir="ltr" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">From:</b><span class="x_Apple-converted-space"> </span>Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>><br class="">
<b class="">Sent:</b><span class="x_Apple-converted-space"> </span>Sunday, February 28, 2021 11:05 PM<br class="">
<b class="">To:</b><span class="x_Apple-converted-space"> </span>Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>>; For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>>;
Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_Apple-converted-space"> </span>Re: [petsc-dev] Infinite loop in A*B</font>
<div class=""> </div>
</div>
<div dir="ltr" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
The infinite loop in <span class="" style="background-color:rgb(255,255,255); display:inline!important">MatMatMultNumeric_MPIAIJ_MPIDense()</span></div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
for (i=0; i<BN; i+=n) {<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
}</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="background-color:rgb(255,255,255); display:inline!important">is cause by n=contents->workB->cmap->n=0 (line 590 in mpimatmatmult.c)</span></div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="background-color:rgb(255,255,255); display:inline!important">Hong</span></div>
<div id="x_x_appendonsend" class=""></div>
<hr tabindex="-1" class="" style="display:inline-block; width:592.890625px">
<div id="x_x_divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">From:</b><span class="x_Apple-converted-space"> </span>petsc-dev <<a href="mailto:petsc-dev-bounces@mcs.anl.gov" class="">petsc-dev-bounces@mcs.anl.gov</a>>
on behalf of Zhang, Hong via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class="">
<b class="">Sent:</b><span class="x_Apple-converted-space"> </span>Sunday, February 28, 2021 10:33 PM<br class="">
<b class="">To:</b><span class="x_Apple-converted-space"> </span>Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>>; For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_Apple-converted-space"> </span>Re: [petsc-dev] Infinite loop in A*B</font>
<div class=""> </div>
</div>
<div dir="ltr" class="">
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
I can reproduce the hang with </div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
mpiexec -n 2 ./matmatmult</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
It seems in an infinite loop of calling MatDensePlaceArray() from </div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
#0 MatDensePlaceArray (mat=0xda5c50, array=0xd15e60)
<div class=""> at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:2047</div>
<div class="">#1 0x00007fa0d13bf4f7 in MatDenseGetSubMatrix_SeqDense (A=0xcfb2b0, cbegin=0,</div>
<div class=""> cend=0, v=0xd90370)</div>
<div class=""> at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:2997</div>
<div class="">#2 0x00007fa0d13c574e in MatDenseGetSubMatrix (A=0xcfb2b0, cbegin=0, cend=0,</div>
<div class=""> v=0xd90370) at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371</div>
<div class="">#3 0x00007fa0d13db5ce in MatDenseGetSubMatrix_MPIDense (A=0xca5250, cbegin=0,</div>
<div class=""> cend=0, v=0x7ffe87d41de0)</div>
<div class=""> at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:1835</div>
<div class="">#4 0x00007fa0d13c574e in MatDenseGetSubMatrix (A=0xca5250, cbegin=0, cend=0,</div>
<div class=""> v=0x7ffe87d41de0)</div>
<div class=""> at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371</div>
<div class="">#5 0x00007fa0d179c2fa in MatMatMultNumeric_MPIAIJ_MPIDense (A=0xc55490,</div>
<div class=""> B=0xca5250, C=0xd282b0)</div>
<div class=""> at /home/hongsu/soft/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:593</div>
<div class="">#6 0x00007fa0d1181331 in MatProductNumeric_AB (mat=0xd282b0)</div>
<div class=""> at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:567</div>
<div class="">#7 0x00007fa0d1182c14 in MatProductNumeric (mat=0xd282b0)</div>
<div class=""> at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:679</div>
<div class="">#8 0x00007fa0d115ef69 in MatProduct_Private (A=0xc55490, B=0xca5250,</div>
<div class=""> scall=MAT_INITIAL_MATRIX, fill=-2, ptype=MATPRODUCT_AB, C=0x7ffe87d42018)</div>
<div class=""> at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9405</div>
<div class="">---Type <return> to continue, or q <return> to quit---</div>
<div class="">#9 0x00007fa0d115f274 in MatMatMult (A=0xc55490, B=0xca5250, scall=MAT_INITIAL_MATRIX, fill=-2,</div>
<div class=""> C=0x7ffe87d42018) at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9445</div>
#10 0x000000000040130a in main (argc=2, argv=0x7ffe87d42108) at ex1.c:20<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
I'll try to figure out what is going on. If anyone has a clue, please help. The above stack comes from 'release' branch.</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
Hong</div>
<div id="x_x_x_appendonsend" class=""></div>
<hr tabindex="-1" class="" style="display:inline-block; width:592.890625px">
<div id="x_x_x_divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">From:</b><span class="x_Apple-converted-space"> </span>petsc-dev <<a href="mailto:petsc-dev-bounces@mcs.anl.gov" class="">petsc-dev-bounces@mcs.anl.gov</a>>
on behalf of Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>><br class="">
<b class="">Sent:</b><span class="x_Apple-converted-space"> </span>Sunday, February 28, 2021 4:17 PM<br class="">
<b class="">To:</b><span class="x_Apple-converted-space"> </span>For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_Apple-converted-space"> </span>[petsc-dev] Infinite loop in A*B</font>
<div class=""> </div>
</div>
<div class="x_x_x_BodyFragment"><font size="2" class=""><span class="" style="font-size:11pt">
<div class="x_x_x_PlainText">Hello,<br class="">
The following MWE loops indefinitely for MPI_Comm_size in {2; 3}.<br class="">
Nothing fancy, just MatAIJ and MatDense.<br class="">
The problem is either in MatMPIDenseScatter() or MatMatMultSymbolic_MPIAIJ_MPIDense(), I believe, so if someone familiar with those routines can figure out a hot fix, I’m all ears.<br class="">
I could of course switch to a MatMult(), but the same infinite loop happens in another more complex code with<br class="">
A = rows=8, cols=35212<br class="">
B = rows=35212, cols=9<br class="">
So I’ll need a fix eventually.<br class="">
<br class="">
Thanks,<br class="">
Pierre</div>
</span></font></div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>