<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
If each block is sequential, try replace <span style="color:rgb(50, 49, 48);font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">SuperLU_DIST<span> with <span style="background-color:rgb(255, 255, 255);display:inline !important">SuperLU,
which would be more robust. You may also try MUMPS LU.</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color:rgb(50, 49, 48);font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important"><span><span style="background-color:rgb(255, 255, 255);display:inline !important">Hong</span></span></span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> petsc-users <petsc-users-bounces@mcs.anl.gov> on behalf of Barry Smith <bsmith@petsc.dev><br>
<b>Sent:</b> Saturday, June 11, 2022 9:45 AM<br>
<b>To:</b> Jorti, Zakariae <zjorti@lanl.gov><br>
<b>Cc:</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov>; Tang, Xianzhu <xtang@lanl.gov><br>
<b>Subject:</b> Re: [petsc-users] [EXTERNAL] Question about SuperLU</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space"><br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 10, 2022, at 7:30 PM, Jorti, Zakariae <<a href="mailto:zjorti@lanl.gov" class="">zjorti@lanl.gov</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div id="x_divtagdefaultwrapper" dir="ltr" class="" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-size:12pt; font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div class="">Hi,<span class="x_Apple-converted-space"> </span><br class="">
<br class="">
<div class="" style="margin-top:0px; margin-bottom:0px">Thank you all for your answers.<span class="x_Apple-converted-space"> </span><br class="">
I have tried your suggestions and here is what I found.<span class="x_Apple-converted-space"> </span><br class="">
Barry you were right about the first case. But in the second case, I am not using a Schur fieldsplit but a multiplicative fieldsplit : -fieldsplit_TEBV_fieldsplit_EBV_pc_type fieldsplit -fieldsplit_TEBV_fieldsplit_EBV_pc_fieldsplit_type multiplicative<span class="x_Apple-converted-space"> </span><br class="">
</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
The previous email indicated</div>
<div><br class="">
</div>
<div>
<blockquote type="cite" class="" style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
<div class="">
<div id="x_divtagdefaultwrapper" dir="ltr" class="" style="font-size:12pt">
<div class="">b) a Schur-based fieldsplit preconditioner that uses SuperLU for both V and B blocks:<br class="">
-fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_ksp_type gmres -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_pc_type fieldsplit -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_pc_fieldsplit_type schur -</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
which means there is a Schur complement PC inside the multiplicative so my explanation that the Schur complement "saves" the problem by passing into SuperLU_DIST a non-singular matrix that is some approximation to the Schur complement could be true.</div>
<div><br class="">
</div>
<div>
<blockquote type="cite" class="">
<div id="x_divtagdefaultwrapper" dir="ltr" class="" style="font-size:12pt; font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div class="">
<div class="" style="margin-top:0px; margin-bottom:0px">Then I tried this flag -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_ksp_view_pmat binary:BVmat:binary_matlab and checked the resulting matrix in Matlab. <br class="">
For Matlab, this is a full rank matrix, and the LU factorization there was carried out without any issues.</div>
<div class="" style="margin-top:0px; margin-bottom:0px">I also outputted the BV block directly from the Jacobian matrix. <br class="">
Once again, according to Matlab, it is a full rank matrix and it computes the LU factorization without any problem.</div>
</div>
</div>
</blockquote>
<div class="">
<div dir="ltr" class="" style="font-size:12pt; font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div class="">
<div class="" style="margin-top:0px; margin-bottom:0px"><br class="">
</div>
<div class="" style="margin-top:0px; margin-bottom:0px"> The BV matrix you saved into Matlab is a "block" matrix where the first block is B and the second block V (presumably both the same size). Can you, in Matlab, extract the two blocks separated and examine
them (via say spy) and also have Matlab factor each of them separately? In your failed fieldsplit case SuperLU_DIST is factoring each of these matrices separately which could produce a zero pivot that would not occur when the larger matrix (of both blocks)
is factored together. Let's see what happens with Matlab's solver.</div>
<div class="" style="margin-top:0px; margin-bottom:0px"><br class="">
</div>
<div class="" style="margin-top:0px; margin-bottom:0px"> It looks like you are running on one rank? If the above process is not informative this is what you do next. </div>
<div class="" style="margin-top:0px; margin-bottom:0px"><br class="">
</div>
<div class="" style="margin-top:0px; margin-bottom:0px"> Use PetscBinaryWrite() from Matlab to save each of the two blocks (one for B and one for V) to two files. Then use a simple standalone PETSc code, say src/ksp/ksp/tutorials/ex10.c to read each of the
files and use SuperLU_DIST directly on each of the two linear systems. This will, at least to my understanding, result in the exact same SuperLU_DIST solves that you get with the failed use of PCFIELDSPLIT. If they succeed or fail will be very informative.</div>
<div class="" style="margin-top:0px; margin-bottom:0px"><br class="">
</div>
<div class="" style="margin-top:0px; margin-bottom:0px"> Barry</div>
<div class="" style="margin-top:0px; margin-bottom:0px"><br class="">
</div>
<div class="" style="margin-top:0px; margin-bottom:0px"><br class="">
</div>
</div>
</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div id="x_divtagdefaultwrapper" dir="ltr" class="" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-size:12pt; font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div class="">
<div class="" style="margin-top:0px; margin-bottom:0px"><br class="">
So, there should not be any Schur complement approximation Sp.<span class="x_Apple-converted-space"> </span><br class="">
<br class="">
When I ran a test with -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_ksp_view_pre -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_ksp_error_if_not_converged, I got this error:<span class="x_Apple-converted-space"> </span><br class="">
</div>
<div class="" style="margin-top:0px; margin-bottom:0px"><br class="">
</div>
<div class=""> 0 SNES Function norm 6.368031218939e-02<span class="x_Apple-converted-space"> </span><br class="">
0 KSP Residual norm 6.368031218939e-02<span class="x_Apple-converted-space"> </span><br class="">
Linear fieldsplit_ni_ solve converged due to CONVERGED_ATOL iterations 0<br class="">
Linear fieldsplit_TEBV_fieldsplit_tau_ solve converged due to CONVERGED_RTOL iterations 1<br class="">
Linear fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_EP_ solve converged due to CONVERGED_RTOL iterations 3<br class="">
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br class="">
[0]PETSC ERROR: Zero pivot in LU factorization:<span class="x_Apple-converted-space"> </span><a href="https://petsc.org/release/faq/#zeropivot" class="">https://petsc.org/release/faq/#zeropivot</a><br class="">
[0]PETSC ERROR: Zero pivot in row 1658<br class="">
[0]PETSC ERROR: See<span class="x_Apple-converted-space"> </span><a href="https://petsc.org/release/faq/" class="">https://petsc.org/release/faq/</a><span class="x_Apple-converted-space"> </span>for trouble shooting.<br class="">
[0]PETSC ERROR: Petsc Development GIT revision: v3.16.3-751-g2f43bd9bc3 GIT Date: 2022-01-26 22:34:02 -0600<br class="">
[0]PETSC ERROR: ./main on a macx named<span class="x_Apple-converted-space"> </span><a href="http://pn2032683.lanl.gov/" class="">pn2032683.lanl.gov</a><span class="x_Apple-converted-space"> </span>by zjorti Fri Jun 10 16:17:35 2022<br class="">
[0]PETSC ERROR: Configure options PETSC_ARCH=macx --with-fc=0 --with-mpi-dir=/Users/zjorti/.brew --download-hypre --with-debugging=0 --with-cxx-dialect=C++11 --download-superlu_dist --download-parmetis --download-metis --download-ptscotch --download-cmake</div>
<br class="">
<br class="">
<p class="" style="margin-top:0px; margin-bottom:0px"></p>
<div class="" style="margin-top:0px; margin-bottom:0px">Then I tried this flag -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_ksp_view_pmat binary:BVmat:binary_matlab and checked the resulting matrix in Matlab.<span class="x_Apple-converted-space"> </span><br class="">
For Matlab, this is a full rank matrix, and the LU factorization there was carried out without any issues.</div>
<div class="" style="margin-top:0px; margin-bottom:0px">I also outputted the BV block directly from the Jacobian matrix.<span class="x_Apple-converted-space"> </span><br class="">
Once again, according to Matlab, it is a full rank matrix and it computes the LU factorization without any problem.</div>
</div>
<br class="">
<br class="">
<div class="" style="">
<hr tabindex="-1" class="" style="display:inline-block; width:1330.828125px">
<div id="x_divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">From:</b><span class="x_Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>><br class="">
<b class="">Sent:</b><span class="x_Apple-converted-space"> </span>Friday, June 10, 2022 7:32 AM<br class="">
<b class="">To:</b><span class="x_Apple-converted-space"> </span>Jorti, Zakariae<br class="">
<b class="">Cc:</b><span class="x_Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>; Tang, Xianzhu<br class="">
<b class="">Subject:</b><span class="x_Apple-converted-space"> </span>[EXTERNAL] Re: [petsc-users] Question about SuperLU</font>
<div class=""> </div>
</div>
<div class="">
<div class=""><br class="">
</div>
It is difficult to tell exactly how the preconditioner is being formed with the information below it looks like in the
<div class=""><br class="">
</div>
<div class="">first case: the original B diagonal block and V diagonal block of the matrix are being factored separately with SuperLU_DIST </div>
<div class=""><br class="">
</div>
<div class="">second case: the B block is factored with SuperLU_DIST and an explicit approximation to a Schur complement of the V block (Schur complement on eliminating the B block) is formed using "Preconditioner for the Schur complement formed from Sp, an
assembled approximation to S, which uses A00's %sdiagonal's inverse" (this is the printout from a KSPView() for this part of the preconditioner). </div>
<div class=""><br class="">
</div>
<div class=""> My guess is you have a "Stokes"-like problem where the V block is identically 0 so, of course, the SuperLU_DIST will fail on it. But the approximation of the Schur complement onto that block is not singular so SuperLU_DIST has no trouble. If
I am wrong and the V block is not identically 0 then it may be singular (or possibly, but less likely just badly order) so that SuperLU_DIST encounters a zero pivot.</div>
<div class=""><br class="">
</div>
<div class=""> You can run with -ksp_view_pre to have the KSP print the KSP solver algorithm details BEFORE the linear solve (hence they would get printed despite your failed solve). That would be useful to see exactly what your preconditioner is.</div>
<div class=""><br class="">
</div>
<div class=""> You can use -ksp_view_pmat (with appropriate prefix) to display the matrix that is going to be factored. Thus you can quickly verify what V is.</div>
<div class=""><br class="">
</div>
<div class=""> If you run with -ksp_error_if_not_converged then the solver will stop exactly when the zero pivot is encountered; this would include some information from SuperLU_DIST which might include the row number etc. </div>
<div class=""><br class="">
</div>
<div class=""> Notes on PETSc improvements needed. </div>
<div class=""><br class="">
</div>
<div class="">1) The man page for KSPCheckSolve() is terribly misleading</div>
<div class=""><br class="">
</div>
<div class="">2) It would be nice to have a view that displayed the nested fieldsplit preconditioners more clearly</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 9, 2022, at 5:19 PM, Jorti, Zakariae via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div id="x_divtagdefaultwrapper" dir="ltr" class="" style="font-style:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-size:12pt; font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p class="" style="margin-top:0px; margin-bottom:0px"></p>
<div class="">Hi,<span class="x_Apple-converted-space"> </span><br class="">
<br class="">
I am solving non-linear problem that has 5 unknowns {ni, T, E, B, V}, and for the preconditioning part, I am using a FieldSplit preconditioner. At the last fieldsplit/level, we are left with a {B,V} block that tried to precondition in 2 different ways:<span class="x_Apple-converted-space"> </span><br class="">
a) SuperLU:<span class="x_Apple-converted-space"> </span><br class="">
-fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_ksp_type preonly -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_pc_type lu -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_pc_factor_mat_solver_type superlu_dist <br class="">
b) a Schur-based fieldsplit preconditioner that uses SuperLU for both V and B blocks:<br class="">
-fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_ksp_type gmres -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_pc_type fieldsplit -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_pc_fieldsplit_type schur -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_pc_fieldsplit_schur_precondition
selfp -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_fieldsplit_B_ksp_type preonly -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_fieldsplit_B_pc_type lu -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_fieldsplit_B_pc_factor_mat_solver_type superlu_dist -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_fieldsplit_V_ksp_type
preonly -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_fieldsplit_V_pc_type lu -fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_fieldsplit_V_pc_factor_mat_solver_type superlu_dist<br class="">
<br class="">
Option a) yields the following error:<span class="x_Apple-converted-space"> </span><br class="">
" Linear fieldsplit_ni_ solve converged due to CONVERGED_ATOL iterations 0<br class="">
Linear fieldsplit_TEBV_fieldsplit_tau_ solve converged due to CONVERGED_RTOL iterations 1<br class="">
Linear fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_EP_ solve converged due to CONVERGED_RTOL iterations 5<br class="">
Linear fieldsplit_TEBV_fieldsplit_EBV_fieldsplit_BV_ solve did not converge due to DIVERGED_PC_FAILED iterations 0<br class="">
PC failed due to FACTOR_NUMERIC_ZEROPIVOT "<span class="x_Apple-converted-space"> </span><br class="">
whereas options b) seems to be working well.<span class="x_Apple-converted-space"> </span><br class="">
Is it possible that the SuperLU on the {V,B} block uses a reordering that introduces a zero pivot or could there be another explanation for this error?</div>
<br class="">
Many thanks.<span class="x_Apple-converted-space"> </span><br class="">
Best,<br class="">
<br class="">
Zakariae</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>