<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 1 Mar 2021, at 6:29 AM, Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="color: rgb(32, 31, 30); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important;" class="">Pierre,</span><br class=""></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font color="#201f1e" class=""><span style="font-size: 14.6667px;" class="">This is a bug in MatMatMultSymbolic_MPIAIJ_MPIDense() during optimization of block column size of B. Run your code with</span></font></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font color="#201f1e" class=""><span style="font-size: 14.6667px;" class="">'-matmatmult_Bbn 1', the infinite loop should not occur.</span></font></div></div></blockquote><div><br class=""></div><div>Thanks Hong, I can confirm this option makes the more complex use case run smoothly as well. </div><br class=""><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font color="#201f1e" class=""><span style="font-size: 14.6667px;" class="">I'll try to figure out a fix tomorrow.<br class=""></span></font></div></div></blockquote><div><br class=""></div><div>Great.</div><div><br class=""></div><div>Thanks,</div><div>Pierre</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font color="#201f1e" class=""><span style="font-size: 14.6667px;" class="">Hong</span></font></div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 592.890625px;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Sunday, February 28, 2021 11:05 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>>; For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>>; Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-dev] Infinite loop in A*B</font><div class=""> </div></div><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">The infinite loop in <span style="background-color: rgb(255, 255, 255); display: inline !important;" class="">MatMatMultNumeric_MPIAIJ_MPIDense()</span></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">for (i=0; i<BN; i+=n) {<br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">}</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="background-color: rgb(255, 255, 255); display: inline !important;" class="">is cause by n=contents->workB->cmap->n=0 (line 590 in mpimatmatmult.c)</span></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="background-color: rgb(255, 255, 255); display: inline !important;" class="">Hong</span></div><div id="x_appendonsend" class=""></div><hr tabindex="-1" style="display: inline-block; width: 592.890625px;" class=""><div id="x_divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>petsc-dev <<a href="mailto:petsc-dev-bounces@mcs.anl.gov" class="">petsc-dev-bounces@mcs.anl.gov</a>> on behalf of Zhang, Hong via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Sunday, February 28, 2021 10:33 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>>; For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-dev] Infinite loop in A*B</font><div class=""> </div></div><div dir="ltr" class=""><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">I can reproduce the hang with </div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">mpiexec -n 2 ./matmatmult</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">It seems in an infinite loop of calling MatDensePlaceArray() from </div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">#0 MatDensePlaceArray (mat=0xda5c50, array=0xd15e60)<div class=""> at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:2047</div><div class="">#1 0x00007fa0d13bf4f7 in MatDenseGetSubMatrix_SeqDense (A=0xcfb2b0, cbegin=0,</div><div class=""> cend=0, v=0xd90370)</div><div class=""> at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:2997</div><div class="">#2 0x00007fa0d13c574e in MatDenseGetSubMatrix (A=0xcfb2b0, cbegin=0, cend=0,</div><div class=""> v=0xd90370) at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371</div><div class="">#3 0x00007fa0d13db5ce in MatDenseGetSubMatrix_MPIDense (A=0xca5250, cbegin=0,</div><div class=""> cend=0, v=0x7ffe87d41de0)</div><div class=""> at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:1835</div><div class="">#4 0x00007fa0d13c574e in MatDenseGetSubMatrix (A=0xca5250, cbegin=0, cend=0,</div><div class=""> v=0x7ffe87d41de0)</div><div class=""> at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371</div><div class="">#5 0x00007fa0d179c2fa in MatMatMultNumeric_MPIAIJ_MPIDense (A=0xc55490,</div><div class=""> B=0xca5250, C=0xd282b0)</div><div class=""> at /home/hongsu/soft/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:593</div><div class="">#6 0x00007fa0d1181331 in MatProductNumeric_AB (mat=0xd282b0)</div><div class=""> at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:567</div><div class="">#7 0x00007fa0d1182c14 in MatProductNumeric (mat=0xd282b0)</div><div class=""> at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:679</div><div class="">#8 0x00007fa0d115ef69 in MatProduct_Private (A=0xc55490, B=0xca5250,</div><div class=""> scall=MAT_INITIAL_MATRIX, fill=-2, ptype=MATPRODUCT_AB, C=0x7ffe87d42018)</div><div class=""> at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9405</div><div class="">---Type <return> to continue, or q <return> to quit---</div><div class="">#9 0x00007fa0d115f274 in MatMatMult (A=0xc55490, B=0xca5250, scall=MAT_INITIAL_MATRIX, fill=-2,</div><div class=""> C=0x7ffe87d42018) at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9445</div>#10 0x000000000040130a in main (argc=2, argv=0x7ffe87d42108) at ex1.c:20<br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">I'll try to figure out what is going on. If anyone has a clue, please help. The above stack comes from 'release' branch.</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">Hong</div><div id="x_x_appendonsend" class=""></div><hr tabindex="-1" style="display: inline-block; width: 592.890625px;" class=""><div id="x_x_divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>petsc-dev <<a href="mailto:petsc-dev-bounces@mcs.anl.gov" class="">petsc-dev-bounces@mcs.anl.gov</a>> on behalf of Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Sunday, February 28, 2021 4:17 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>[petsc-dev] Infinite loop in A*B</font><div class=""> </div></div><div class="x_x_BodyFragment"><font size="2" class=""><span style="font-size: 11pt;" class=""><div class="x_x_PlainText">Hello,<br class="">The following MWE loops indefinitely for MPI_Comm_size in {2; 3}.<br class="">Nothing fancy, just MatAIJ and MatDense.<br class="">The problem is either in MatMPIDenseScatter() or MatMatMultSymbolic_MPIAIJ_MPIDense(), I believe, so if someone familiar with those routines can figure out a hot fix, I’m all ears.<br class="">I could of course switch to a MatMult(), but the same infinite loop happens in another more complex code with<br class="">A = rows=8, cols=35212<br class="">B = rows=35212, cols=9<br class="">So I’ll need a fix eventually.<br class=""><br class="">Thanks,<br class="">Pierre</div></span></font></div></div></div></div></blockquote></div><br class=""></body></html>