<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><br><div><blockquote type="cite"><div>On 28 Aug 2023, at 6:50 PM, Carl-Johan Thore <carljohanthore@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr"><div>I've tried the new files, and with them, PCFIELDSPLIT now gets set up without crashes (but the setup is significantly slower than for MATAIJ)<br></div></div></div></blockquote><div><br></div><div>I’ll be back from Japan at the end of this week, my schedule is too packed to get anything done in the meantime.</div><div>But I’ll let you know when things are working properly (last I checked, I think it was working, but I may have forgotten about a corner case or two).</div><div>But yes, though one would except things to be faster and less memory intensive with SBAIJ, it’s unfortunately not always the case.</div><div><br></div><div>Thanks,</div><div>Pierre</div><br><blockquote type="cite"><div><div dir="ltr"><div>Unfortunately I still get errors later in the process:<br></div><div><br></div><div>[0]PETSC ERROR: Null argument, when expecting valid pointer<br>[0]PETSC ERROR: Null Pointer: Parameter # 1<br>[0]PETSC ERROR: Petsc Development GIT revision: v3.19.4-1023-ga6d78fcba1d GIT Date: 2023-08-22 20:32:33 -0400<br>[0]PETSC ERROR: Configure options -f --with-fortran-bindings=0 --with-cuda --with-cusp --download-scalapack --download-hdf5 --download-zlib --download-mumps --download-parmetis --download-metis --download-ptscotch --download-hypre --download-spai<br>[0]PETSC ERROR: #1 PetscObjectQuery() at /mnt/c/mathware/petsc/petsc-v3-19-4/src/sys/objects/inherit.c:742<br>[0]PETSC ERROR: #2 MatCreateSubMatrix_MPISBAIJ() at /mnt/c/mathware/petsc/petsc-v3-19-4/src/mat/impls/sbaij/mpi/mpisbaij.c:1414<br>[0]PETSC ERROR: #3 MatCreateSubMatrix() at /mnt/c/mathware/petsc/petsc-v3-19-4/src/mat/interface/matrix.c:8476<br>[0]PETSC ERROR: #4 PCSetUp_FieldSplit() at /mnt/c/mathware/petsc/petsc-v3-19-4/src/ksp/pc/impls/fieldsplit/fieldsplit.c:826<br>[0]PETSC ERROR: #5 PCSetUp() at /mnt/c/mathware/petsc/petsc-v3-19-4/src/ksp/pc/interface/precon.c:1069<br>[0]PETSC ERROR: #6 KSPSetUp() at /mnt/c/mathware/petsc/petsc-v3-19-4/src/ksp/ksp/interface/itfunc.c:415</div><div><br></div><div>The code I'm running here works without any problems for MATAIJ. To run it with MATSBAIJ I've simply used the command-line option</div><div>-dm_mat_type sbaij</div><br><div><br></div><div>Kind regards,</div><div>Carl-Johan<br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 26, 2023 at 5:21 PM Pierre Jolivet via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div><br><blockquote type="cite"><div>On 27 Aug 2023, at 12:14 AM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" target="_blank">carl-johan.thore@liu.se</a>> wrote:</div><br><div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">“Well, your A00 and A11 will possibly be SBAIJ also, so you’ll end up with the same issue.”<u></u><u></u></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">I’m not sure I follow. Does PCFIELDSPLIT extract further submatrices from these blocks, or is there<u></u><u></u></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">somewhere else in the code that things will go wrong?</div></div></div></blockquote><div><br></div><div>Ah, no, you are right, in that case it should work.</div><br><blockquote type="cite"><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">For the MATNEST I was thinking to get some savings from the block-symmetry at least<u></u><u></u></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">even if symmetry in A00 and A11 cannot be exploited; using SBAIJ for them would just be a<u></u><u></u></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">(pretty big) bonus.<u></u><u></u></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">“I’ll rebase on top of main and try to get it integrated if it could be useful to you (but I’m traveling<u></u><u></u></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">right now so everything gets done more slowly, sorry).”<u></u><u></u></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">Sound great, Thanks again!</div></div></blockquote><div><br></div><div>The MR is there <a href="https://gitlab.com/petsc/petsc/-/merge_requests/6841" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/6841</a>.</div><div>I need to add a new code path in MatCreateRedundantMatrix() to make sure the resulting Mat is indeed SBAIJ, but that is orthogonal to the PCFIELDSPLIT issue.</div><div>The branch should be usable in its current state.</div><div><br></div><div>Thanks,</div><div>Pierre</div><div><br></div><blockquote type="cite"><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div><div><div style="border-width:1pt medium medium;border-style:solid none none;border-color:rgb(225,225,225) currentcolor currentcolor;padding:3pt 0cm 0cm"><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"><b>From:</b><span> </span>Pierre Jolivet <<a href="mailto:pierre.jolivet@lip6.fr" target="_blank">pierre.jolivet@lip6.fr</a>><span> </span><br><b>Sent:</b><span> </span>Saturday, August 26, 2023 4:36 PM<br><b>To:</b><span> </span>Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" target="_blank">carl-johan.thore@liu.se</a>><br><b>Cc:</b><span> </span>Carl-Johan Thore <<a href="mailto:carljohanthore@gmail.com" target="_blank">carljohanthore@gmail.com</a>>; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br><b>Subject:</b><span> </span>Re: [petsc-users] PCFIELDSPLIT with MATSBAIJ<u></u><u></u></div></div></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"><br><br><u></u><u></u></div><blockquote style="margin-top:5pt;margin-bottom:5pt"><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">On 26 Aug 2023, at 11:16 PM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color:blue;text-decoration:underline" target="_blank">carl-johan.thore@liu.se</a>> wrote:<u></u><u></u></div></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div><div><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">"(Sadly) MATSBAIJ is extremely broken, in particular, it cannot be used to retrieve rectangular blocks in MatCreateSubMatrices, thus you cannot get the A01 and A10 blocks in PCFIELDSPLIT.<br>I have a branch that fixes this, but I haven’t rebased in a while (and I’m AFK right now), would you want me to rebase and give it a go, or must you stick to a release tarball?"<br><br>Ok, would be great if you could look at this! I don't need to stick to any particular branch.<br><br>Do you think MATNEST could be an alternative here?<u></u><u></u></div></div></div></blockquote><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div></div><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">Well, your A00 and A11 will possibly be SBAIJ also, so you’ll end up with the same issue.<u></u><u></u></div></div><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">I’m using both approaches (monolithic SBAIJ or Nest + SBAIJ), it was crashing but I think it was thoroughly fixed in <a href="https://gitlab.com/petsc/petsc/-/commits/jolivet/feature-matcreatesubmatrices-rectangular-sbaij/" style="color:blue;text-decoration:underline" target="_blank">https://gitlab.com/petsc/petsc/-/commits/jolivet/feature-matcreatesubmatrices-rectangular-sbaij/</a><u></u><u></u></div></div><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">It is ugly code on top of ugly code, so I didn’t try to get it integrated and just used the branch locally, and then moved to some other stuff.<u></u><u></u></div></div><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">I’ll rebase on top of main and try to get it integrated if it could be useful to you (but I’m traveling right now so everything gets done more slowly, sorry).<u></u><u></u></div></div><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div></div><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">Thanks,<u></u><u></u></div></div><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">Pierre<u></u><u></u></div></div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"><br><br><u></u><u></u></div><blockquote style="margin-top:5pt;margin-bottom:5pt"><div><div><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">My matrix is<br>[A00 A01;<br>A01^t A11]<br>so perhaps with MATNEST I can make use of the block-symmetry at least, and then use MATSBAIJ for<span> </span><br>A00 and A11 if it's possible to combine matrix types which the manual seems to imply.<span> </span><br><br>Kind regards<br>Carl-Johan<br><br><br><br><u></u><u></u></div><blockquote style="margin-top:5pt;margin-bottom:5pt"><div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">On 26 Aug 2023, at 10:09 PM, Carl-Johan Thore <<a href="mailto:carljohanthore@gmail.com" style="color:blue;text-decoration:underline" target="_blank">carljohanthore@gmail.com</a>> wrote:<br><br>Hi,<br><br>I'm trying to use PCFIELDSPLIT with MATSBAIJ in PETSc 3.19.4.<span> </span><br>According to the manual "[t]he fieldsplit preconditioner cannot<span> </span><br>currently be used with the MATBAIJ or MATSBAIJ data formats if the<span> </span><br>blocksize is larger than 1". Since my blocksize is exactly 1 it would seem that I can use PCFIELDSPLIT. But this fails with "PETSC ERROR: For symmetric format, iscol must equal isrow"<br>from MatCreateSubMatrix_MPISBAIJ. Tracing backwards one ends up in<span> </span><br>fieldsplit.c at<br><br>/* extract the A01 and A10 matrices */ ilink = jac->head;<span> </span><br>PetscCall(ISComplement(ilink->is_col, rstart, rend, &ccis)); if<span> </span><br>(jac->offdiag_use_amat) { PetscCall(MatCreateSubMatrix(pc->mat,<span> </span><br>ilink->is, ccis, MAT_INITIAL_MATRIX, &jac->B)); } else {<br> PetscCall(MatCreateSubMatrix(pc->pmat, ilink->is, ccis,<span> </span><br>MAT_INITIAL_MATRIX, &jac->B)); }<br><br>This, since my A01 and A10 are not square, seems to explain why iscol is not equal to isrow.<br>From this I gather that it is in general NOT possible to use<span> </span><br>PCFIELDSPLIT with MATSBAIJ even with block size 1?<br><br>Kind regards,<br>Carl-Johan</div></blockquote></div></div></blockquote></div></div></blockquote></div><br></div></blockquote></div></div>
</div></blockquote></div><br></body></html>