<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif;font-size:14px;background-color:rgb(255, 255, 255);display:inline !important">Pierre<b>,</b></span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This<b> </b>would be helpful to users. Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hong</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Pierre Jolivet <pierre@joliv.et><br>
<b>Sent:</b> Thursday, March 18, 2021 10:08 AM<br>
<b>To:</b> Zhang, Hong <hzhang@mcs.anl.gov><br>
<b>Cc:</b> For users of the development version of PETSc <petsc-dev@mcs.anl.gov>; Patrick Sanan <patrick.sanan@gmail.com><br>
<b>Subject:</b> Re: [petsc-dev] MatTransposeMatMult() bug</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">Thanks for the suggestion Hong.
<div class="">I’ve been somehow putting on hold edits in the documentation (src/docs/website/documentation/changes/dev.html notwithstanding) because I’m not sure yet of what will be migrated or not.</div>
<div class="">But I’ll edit the page according to Patrick’s comment.</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Pierre<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 18 Mar 2021, at 4:00 PM, Patrick Sanan <<a href="mailto:patrick.sanan@gmail.com" class="">patrick.sanan@gmail.com</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div class="" style="word-wrap:break-word; line-break:after-white-space">Sorry about the current mess but that page is halfway migrated, so any updates should go here:
<div class=""><a href="https://docs.petsc.org/en/main/install/externalsoftware_documentation/" class="">https://docs.petsc.org/en/main/install/externalsoftware_documentation/</a></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">Am 18.03.2021 um 15:22 schrieb Zhang, Hong via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>>:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline!important">Pierre,</span><br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline!important">This is an external package to petsc. Shall it be listed at</span></div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline!important"><a href="https://www.mcs.anl.gov/petsc/miscellaneous/external.html" id="LPlnk" class="">https://www.mcs.anl.gov/petsc/miscellaneous/external.html</a>?</span></div>
<div class="x__EReadonly_1 x__EType_OWALinkPreview x__Entity x__EId_OWALinkPreview" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div id="LPBorder_GTaHR0cHM6Ly93d3cubWNzLmFubC5nb3YvcGV0c2MvbWlzY2VsbGFuZW91cy9leHRlcm5hbC5odG1s" class="x_LPBorder639529" style="width:800px; margin-top:16px; margin-bottom:16px; max-width:800px; min-width:424px">
<table id="LPContainer639529" role="presentation" class="" style="padding:12px 36px 12px 12px; width:750px; border:1px solid rgb(200,200,200); border-top-left-radius:2px; border-top-right-radius:2px; border-bottom-right-radius:2px; border-bottom-left-radius:2px">
<tbody class="">
<tr valign="top" class="" style="border-spacing:0px">
<td class="" style="width:744px">
<div id="LPTitle639529" class="" style="font-size:21px; font-weight:300; margin-right:8px; font-family:wf_segoe-ui_light,"Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; margin-bottom:12px">
<a target="_blank" id="LPUrlAnchor639529" href="https://www.mcs.anl.gov/petsc/miscellaneous/external.html" class="" style="text-decoration:none">PETSc: External Software - anl.gov</a></div>
<div id="LPDescription639529" class="" style="font-size:14px; max-height:100px; color:rgb(102,102,102); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; margin-bottom:12px; margin-right:8px; overflow:hidden">
PETSc interfaces to the following optional external software (installing packages) (manual pages):. AMD - Approximate minimum degree orderings.; BLAS and LAPACK; Chaco - a graph partitioning package.; ESSL - IBM's math library for fast sparse direct LU factorization.
FFTW - Fastest Fourier Transform in the West, developed at MIT by Matteo Frigo and Steven G. Johnson.</div>
<div id="LPMetadata639529" class="" style="font-size:14px; font-weight:400; color:rgb(166,166,166); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif">
<a href="http://www.mcs.anl.gov/" class="">www.mcs.anl.gov</a></div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
Hong</div>
<div id="x_appendonsend" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
</div>
<hr tabindex="-1" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; display:inline-block; width:848.671875px">
<span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="x_divRplyFwdMsg" dir="ltr" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">From:</b><span class="x_Apple-converted-space"> </span>Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>><br class="">
<b class="">Sent:</b><span class="x_Apple-converted-space"> </span>Thursday, March 18, 2021 1:16 AM<br class="">
<b class="">To:</b><span class="x_Apple-converted-space"> </span>Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>><br class="">
<b class="">Cc:</b><span class="x_Apple-converted-space"> </span>For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_Apple-converted-space"> </span>Re: [petsc-dev] MatTransposeMatMult() bug</font>
<div class=""> </div>
</div>
<div class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<a href="https://www.sciencedirect.com/science/article/abs/pii/S0898122121000055" class="">https://www.sciencedirect.com/science/article/abs/pii/S0898122121000055</a>
<div class=""><a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPHPDDM.html" class="">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPHPDDM.html</a></div>
<div class=""><a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHPDDM.html" class="">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHPDDM.html</a></div>
<div class="">
<div class="">I need to update the PETSc user manual though, specifically with respect to systems with multiple right-hand sides.</div>
</div>
<div class="">But don’t worry, Stefano has sorted the bug out, which was due to a faulty MatSetFromOptions() in MatMAIJ, used by MatTransposeMatMult().</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Pierre<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 17 Mar 2021, at 11:21 PM, Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>> wrote:</div>
<br class="x_x_Apple-interchange-newline">
<div class="">
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
What is <span class="" style="font-size:14px; background-color:rgb(255,255,255); display:inline!important">hpddm? I do not see its document.</span></div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="font-size:14px; background-color:rgb(255,255,255); display:inline!important">Hong</span></div>
<div id="x_x_appendonsend" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<hr tabindex="-1" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; display:inline-block; width:592.890625px">
<span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="x_x_divRplyFwdMsg" dir="ltr" class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">From:</b><span class="x_x_Apple-converted-space"> </span>Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>><br class="">
<b class="">Sent:</b><span class="x_x_Apple-converted-space"> </span>Wednesday, March 17, 2021 2:49 PM<br class="">
<b class="">To:</b><span class="x_x_Apple-converted-space"> </span>Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" class="">hzhang@mcs.anl.gov</a>><br class="">
<b class="">Cc:</b><span class="x_x_Apple-converted-space"> </span>Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>>; For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_x_Apple-converted-space"> </span>Re: [petsc-dev] MatTransposeMatMult() bug</font>
<div class=""> </div>
</div>
<div class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div dir="ltr" class="">
<div dir="ltr" class="">On Wed, Mar 17, 2021 at 3:27 PM Zhang, Hong via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>> wrote:<br class="">
</div>
<div class="x_x_x_gmail_quote">
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div dir="ltr" class="">
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline">Pierre,</span><br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline">Do you mean a possible bug in C=<span class="" style="background-color:rgb(255,255,255); display:inline">AtB <span class="" style="background-color:rgb(255,255,255); display:inline">MatTransposeMatMult()?</span></span></span></div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline"><span class="" style="background-color:rgb(255,255,255); display:inline"><span class="" style="background-color:rgb(255,255,255); display:inline">Can
you provide a stand-alone test without hpddm that reproduces this error? </span></span></span></div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Hong, you should be able to just configure with --download-hpddm and then run that ex76 test.</div>
<div class=""><br class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div dir="ltr" class="">
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">
<span class="" style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline"><span class="" style="background-color:rgb(255,255,255); display:inline"><span class="" style="background-color:rgb(255,255,255); display:inline">Hong</span></span></span></div>
<div id="x_x_x_gmail-m_-5972319960547956757appendonsend" class=""></div>
<hr class="" style="display:inline-block; width:580.859375px">
<div id="x_x_x_gmail-m_-5972319960547956757divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">From:</b><span class="x_x_Apple-converted-space"> </span>petsc-dev <<a href="mailto:petsc-dev-bounces@mcs.anl.gov" target="_blank" class="">petsc-dev-bounces@mcs.anl.gov</a>>
on behalf of Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank" class="">pierre@joliv.et</a>><br class="">
<b class="">Sent:</b><span class="x_x_Apple-converted-space"> </span>Wednesday, March 17, 2021 4:31 AM<br class="">
<b class="">To:</b><span class="x_x_Apple-converted-space"> </span>For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank" class="">petsc-dev@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="x_x_Apple-converted-space"> </span>[petsc-dev] MatTransposeMatMult() bug</font>
<div class=""> </div>
</div>
<div class="">Hello,
<div class="">While trying out Stefano’s PCApplyMat_MG() code (*), we stumbled upon weird numerical errors when reusing a Mat for both MatProduct_AB and MatProduct_AtB.</div>
<div class="">This reminded me that there has been a long-standing issue with MatTransposeMatMult(), see <a href="https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cxx.html#line608" target="_blank" class="">https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cxx.html#line608</a>,
that I never looked into.</div>
<div class="">I’ve now been trying to figure this out, because this has side effects in multiple places (PCMG and PCHPDDM at least), and thus could impact user-code as well?</div>
<div class="">With this commit: <a href="https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aaceba0" target="_blank" class="">https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aaceba0</a></div>
<div class="">+</div>
<div class="">$ mpirun -n 4 src/ksp/ksp/tutorials/ex76 -ksp_converged_reason -pc_type hpddm -pc_hpddm_levels_1_eps_nev 20 -ksp_type preonly -mat_type aij -load_dir ${DATAFILESPATH}/matrices/hpddm/GENEO -rhs 2 -pc_hpddm_coarse_correction balanced -C_input_mattransposematmult
-D_output_mattransposematmult</div>
<div class="">I’m seeing that C is nonzero, but D is full of zeros.</div>
<div class="">
<div class="">Mat Object: 4 MPI processes</div>
<div class=""> type: mpidense</div>
<div class="">5.7098316584361917e-08 1.0159399260517841e-07</div>
<div class="">1.5812349976211856e-07 2.0688121715350138e-07</div>
<div class="">2.4887556933361981e-08 4.8111092300772958e-08</div>
<div class="">1.4606298643602107e-07 1.7213611729839211e-07</div>
</div>
<div class="">[…]</div>
<div class="">
<div class="">Mat Object: 4 MPI processes</div>
<div class=""> type: mpidense</div>
<div class="">0.0000000000000000e+00 0.0000000000000000e+00</div>
<div class="">0.0000000000000000e+00 0.0000000000000000e+00</div>
<div class="">0.0000000000000000e+00 0.0000000000000000e+00</div>
<div class="">0.0000000000000000e+00 0.0000000000000000e+00</div>
</div>
<div class="">[…]</div>
<div class=""><br class="">
</div>
<div class="">If one switches to a MatType which has no MatProduct_AtB implementation with B of type MPIDense (reminder: in that case, the product is computed column-by-column), e.g., -mat_type sbaij, one gets the expected result.</div>
<div class="">
<div class="">Mat Object: 4 MPI processes</div>
<div class=""> type: mpidense</div>
<div class="">7.2003197398135299e-01 9.5191869895699011e-01</div>
<div class="">6.1793966541680234e-02 9.3884397585488877e-01</div>
<div class="">1.0022337823233585e-02 2.4653068080134588e-01</div>
<div class="">1.4463931936094099e-01 8.6111517670701687e-01</div>
</div>
<div class=""><br class="">
</div>
<div class="">Is there a bug somewhere with the MatAIJ implementation, or am I doing something which is not allowed by the MatProduct() machinery?</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Pierre</div>
<div class=""><br class="">
</div>
<div class="">(*) <a href="https://gitlab.com/petsc/petsc/-/merge_requests/3717" target="_blank" class="">https://gitlab.com/petsc/petsc/-/merge_requests/3717</a> </div>
</div>
</div>
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class="x_x_Apple-converted-space"> </span><br class="">
<div dir="ltr" class="x_x_x_gmail_signature">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>