<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><a href="https://www.sciencedirect.com/science/article/abs/pii/S0898122121000055" class="">https://www.sciencedirect.com/science/article/abs/pii/S0898122121000055</a><div class=""><a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPHPDDM.html" class="">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPHPDDM.html</a></div><div class=""><a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHPDDM.html" class="">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHPDDM.html</a></div><div class=""><div class="">I need to update the PETSc user manual though, specifically with respect to systems with multiple right-hand sides.</div></div><div class="">But don’t worry, Stefano has sorted the bug out, which was due to a faulty MatSetFromOptions() in MatMAIJ, used by MatTransposeMatMult().</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Pierre<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 17 Mar 2021, at 11:21 PM, Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">What is <span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14px; background-color: rgb(255, 255, 255); display: inline !important;" class="">hpddm? I do not see its document.</span></div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14px; background-color: rgb(255, 255, 255); display: inline !important;" class="">Hong</span></div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 592.890625px;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, March 17, 2021 2:49 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>><br class=""><b class="">Cc:</b><span class="Apple-converted-space"> </span>Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>>; For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-dev] MatTransposeMatMult() bug</font><div class=""> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div dir="ltr" class=""><div dir="ltr" class="">On Wed, Mar 17, 2021 at 3:27 PM Zhang, Hong via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>> wrote:<br class=""></div><div class="x_gmail_quote"><blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="color: rgb(32, 31, 30); font-size: 15px; background-color: rgb(255, 255, 255); display: inline;" class="">Pierre,</span><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="color: rgb(32, 31, 30); font-size: 15px; background-color: rgb(255, 255, 255); display: inline;" class="">Do you mean a possible bug in C=<span style="background-color: rgb(255, 255, 255); display: inline;" class="">AtB <span style="background-color: rgb(255, 255, 255); display: inline;" class="">MatTransposeMatMult()?</span></span></span></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="color: rgb(32, 31, 30); font-size: 15px; background-color: rgb(255, 255, 255); display: inline;" class=""><span style="background-color: rgb(255, 255, 255); display: inline;" class=""><span style="background-color: rgb(255, 255, 255); display: inline;" class="">Can you provide a stand-alone test without hpddm that reproduces this error? </span></span></span></div></div></blockquote><div class=""><br class=""></div><div class="">Hong, you should be able to just configure with --download-hpddm and then run that ex76 test.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="color: rgb(32, 31, 30); font-size: 15px; background-color: rgb(255, 255, 255); display: inline;" class=""><span style="background-color: rgb(255, 255, 255); display: inline;" class=""><span style="background-color: rgb(255, 255, 255); display: inline;" class="">Hong</span></span></span></div><div id="x_gmail-m_-5972319960547956757appendonsend" class=""></div><hr style="display: inline-block; width: 580.859375px;" class=""><div id="x_gmail-m_-5972319960547956757divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>petsc-dev <<a href="mailto:petsc-dev-bounces@mcs.anl.gov" target="_blank" class="">petsc-dev-bounces@mcs.anl.gov</a>> on behalf of Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank" class="">pierre@joliv.et</a>><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, March 17, 2021 4:31 AM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank" class="">petsc-dev@mcs.anl.gov</a>><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>[petsc-dev] MatTransposeMatMult() bug</font><div class=""> </div></div><div class="">Hello,<div class="">While trying out Stefano’s PCApplyMat_MG() code (*), we stumbled upon weird numerical errors when reusing a Mat for both MatProduct_AB and MatProduct_AtB.</div><div class="">This reminded me that there has been a long-standing issue with MatTransposeMatMult(), see <a href="https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cxx.html#line608" target="_blank" class="">https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cxx.html#line608</a>, that I never looked into.</div><div class="">I’ve now been trying to figure this out, because this has side effects in multiple places (PCMG and PCHPDDM at least), and thus could impact user-code as well?</div><div class="">With this commit: <a href="https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aaceba0" target="_blank" class="">https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aaceba0</a></div><div class="">+</div><div class="">$ mpirun -n 4 src/ksp/ksp/tutorials/ex76 -ksp_converged_reason -pc_type hpddm -pc_hpddm_levels_1_eps_nev 20 -ksp_type preonly -mat_type aij -load_dir ${DATAFILESPATH}/matrices/hpddm/GENEO -rhs 2 -pc_hpddm_coarse_correction balanced -C_input_mattransposematmult -D_output_mattransposematmult</div><div class="">I’m seeing that C is nonzero, but D is full of zeros.</div><div class=""><div class="">Mat Object: 4 MPI processes</div><div class=""> type: mpidense</div><div class="">5.7098316584361917e-08 1.0159399260517841e-07</div><div class="">1.5812349976211856e-07 2.0688121715350138e-07</div><div class="">2.4887556933361981e-08 4.8111092300772958e-08</div><div class="">1.4606298643602107e-07 1.7213611729839211e-07</div></div><div class="">[…]</div><div class=""><div class="">Mat Object: 4 MPI processes</div><div class=""> type: mpidense</div><div class="">0.0000000000000000e+00 0.0000000000000000e+00</div><div class="">0.0000000000000000e+00 0.0000000000000000e+00</div><div class="">0.0000000000000000e+00 0.0000000000000000e+00</div><div class="">0.0000000000000000e+00 0.0000000000000000e+00</div></div><div class="">[…]</div><div class=""><br class=""></div><div class="">If one switches to a MatType which has no MatProduct_AtB implementation with B of type MPIDense (reminder: in that case, the product is computed column-by-column), e.g., -mat_type sbaij, one gets the expected result.</div><div class=""><div class="">Mat Object: 4 MPI processes</div><div class=""> type: mpidense</div><div class="">7.2003197398135299e-01 9.5191869895699011e-01</div><div class="">6.1793966541680234e-02 9.3884397585488877e-01</div><div class="">1.0022337823233585e-02 2.4653068080134588e-01</div><div class="">1.4463931936094099e-01 8.6111517670701687e-01</div></div><div class=""><br class=""></div><div class="">Is there a bug somewhere with the MatAIJ implementation, or am I doing something which is not allowed by the MatProduct() machinery?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Pierre</div><div class=""><br class=""></div><div class="">(*) <a href="https://gitlab.com/petsc/petsc/-/merge_requests/3717" target="_blank" class="">https://gitlab.com/petsc/petsc/-/merge_requests/3717</a> </div></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class="x_gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></body></html>