<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<blockquote type="cite" style="font-family: "Segoe UI Web (West European)", "Segoe UI", -apple-system, "system-ui", Roboto, "Helvetica Neue", sans-serif; font-size: 12px; background-color: rgb(255, 255, 255);">
<div style="font-family:Helvetica;margin:0px;text-decoration:none">
<div style="margin:0px">
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div style="margin:0px">
<div style="margin:0px"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Carl-Johan,</span><br>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><span style="">Thanks for the advice. Our code is well-tested for AIJ. Is SBAIJ inherently significantly slower than AIJ, or is it just that it's not so much used and thus not as developed?</span><br>
</div>
<div>
<div dir="ltr">
<blockquote style="margin-top:0;margin-bottom:0">
<div style="color: rgb(0, 0, 0);"><font face="Calibri, Arial, Helvetica, sans-serif"><span style="color: rgb(88, 93, 237);">SBAIJ only stores upper half triangular part of matrix. When it needs a lower triangular part of entry, it has to jump around searching
for that entry (column search instead of row accessing in AIJ), causing overhead for data-accessing. In parallel computation, it leads to extra inter-processor communication. </span></font></div>
<div style="color: rgb(0, 0, 0);"><font face="Calibri, Arial, Helvetica, sans-serif"><span style="color: rgb(88, 93, 237);">Hong</span></font></div>
</blockquote>
</div>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sat, Aug 26, 2023 at 6:27 PM Zhang, Hong via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" id="OWAf9c38c13-6efd-3813-4aad-0ffacdc5975d" class="OWAAutoLink">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg856845290421164198">
<div dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I would suggest avoiding using SBAIJ matrices, at least in the phase of application code development. We implemented <span style="display: inline; background-color: rgb(255, 255, 255);">SBAIJ<span> for saving storage, not computational efficiency. SBAIJ does
not have as many support as AIJ. After your code works for AIJ, then you may consider taking advantage of smaller storage of SBAIJ (could at cost of communication overhead).</span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="display: inline; background-color: rgb(255, 255, 255);"><span>Hong</span></span></div>
<div id="x_m_856845290421164198appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_856845290421164198divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> petsc-users <<a href="mailto:petsc-users-bounces@mcs.anl.gov" id="OWAd720aacb-d490-99f9-3c4f-8450d6dce9d9" class="OWAAutoLink">petsc-users-bounces@mcs.anl.gov</a>>
on behalf of Pierre Jolivet via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" id="OWAf60f532f-f7db-0772-290b-7ea2b724c499" class="OWAAutoLink">petsc-users@mcs.anl.gov</a>><br>
<b>Sent:</b> Saturday, August 26, 2023 10:20 AM<br>
<b>To:</b> Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" id="OWA1f55fb8b-f532-70c4-9100-bca448f879d9" class="OWAAutoLink">carl-johan.thore@liu.se</a>><br>
<b>Cc:</b> petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" id="OWA96f6345a-e95d-903a-a9b7-184257834262" class="OWAAutoLink">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] PCFIELDSPLIT with MATSBAIJ</font>
<div> </div>
</div>
<div><br>
<div><br>
<blockquote type="cite">
<div>On 27 Aug 2023, at 12:14 AM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" id="OWA0b04908f-1632-5868-b351-496c75e9d1af" class="OWAAutoLink">carl-johan.thore@liu.se</a>> wrote:</div>
<br>
<div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">“Well, your A00 and A11 will possibly be SBAIJ also, so you’ll end up with the same issue.”</div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">I’m not sure I follow. Does PCFIELDSPLIT extract further submatrices from these blocks, or is there</div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">somewhere else in the code that things will go wrong?</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Ah, no, you are right, in that case it should work.</div>
<br>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">For the MATNEST I was thinking to get some savings from the block-symmetry at least</div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">even if symmetry in A00 and A11 cannot be exploited; using SBAIJ for them would just be a</div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">(pretty big) bonus.</div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif"> </div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">“I’ll rebase on top of main and try to get it integrated if it could be useful to you (but I’m traveling</div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">right now so everything gets done more slowly, sorry).”</div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">Sound great, Thanks again!</div>
</div>
</blockquote>
<div><br>
</div>
<div>The MR is there <a href="https://gitlab.com/petsc/petsc/-/merge_requests/6841" data-auth="NotApplicable" id="OWA50da3784-2f49-6463-3c2c-a1daeed90c33" class="OWAAutoLink">https://gitlab.com/petsc/petsc/-/merge_requests/6841</a>.</div>
<div>I need to add a new code path in MatCreateRedundantMatrix() to make sure the resulting Mat is indeed SBAIJ, but that is orthogonal to the PCFIELDSPLIT issue.</div>
<div>The branch should be usable in its current state.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Pierre</div>
<div><br>
</div>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif"> </div>
<div>
<div style="border-width:1pt medium medium; border-style:solid none none; border-color:rgb(225,225,225) currentcolor currentcolor; padding:3pt 0cm 0cm">
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif"><b>From:</b><span> </span>Pierre Jolivet <<a href="mailto:pierre.jolivet@lip6.fr" id="OWA8775c303-cbb7-13c4-dc5e-68b5423899ba" class="OWAAutoLink">pierre.jolivet@lip6.fr</a>><span> </span><br>
<b>Sent:</b><span> </span>Saturday, August 26, 2023 4:36 PM<br>
<b>To:</b><span> </span>Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" id="OWAc2f80929-b959-3813-4c40-509ca2a71495" class="OWAAutoLink">carl-johan.thore@liu.se</a>><br>
<b>Cc:</b><span> </span>Carl-Johan Thore <<a href="mailto:carljohanthore@gmail.com" id="OWA05e397cd-66cb-cd74-777d-ab7476afb84b" class="OWAAutoLink">carljohanthore@gmail.com</a>>;
<a href="mailto:petsc-users@mcs.anl.gov" id="OWAb66c50be-dd83-c858-0c85-6899eac0ae5e" class="OWAAutoLink">
petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b><span> </span>Re: [petsc-users] PCFIELDSPLIT with MATSBAIJ</div>
</div>
</div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif"> </div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif"> </div>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif"><br>
<br>
</div>
<blockquote style="margin-top:5pt; margin-bottom:5pt">
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">On 26 Aug 2023, at 11:16 PM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="text-decoration: underline; color: blue;" id="OWA66f72aee-2d69-3c40-c9b7-9105a5ccbaa9" class="OWAAutoLink">carl-johan.thore@liu.se</a>>
wrote:</div>
</div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif"> </div>
<div>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">"(Sadly) MATSBAIJ is extremely broken, in particular, it cannot be used to retrieve rectangular blocks in MatCreateSubMatrices, thus you cannot get the A01 and A10 blocks in PCFIELDSPLIT.<br>
I have a branch that fixes this, but I haven’t rebased in a while (and I’m AFK right now), would you want me to rebase and give it a go, or must you stick to a release tarball?"<br>
<br>
Ok, would be great if you could look at this! I don't need to stick to any particular branch.<br>
<br>
Do you think MATNEST could be an alternative here?</div>
</div>
</div>
</blockquote>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif"> </div>
</div>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">Well, your A00 and A11 will possibly be SBAIJ also, so you’ll end up with the same issue.</div>
</div>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">I’m using both approaches (monolithic SBAIJ or Nest + SBAIJ), it was crashing but I think it was thoroughly fixed in <a href="https://gitlab.com/petsc/petsc/-/commits/jolivet/feature-matcreatesubmatrices-rectangular-sbaij/" data-auth="NotApplicable" style="text-decoration: underline; color: blue;" id="OWAb35b78d2-e101-9aa9-2ec6-205d6c38439d" class="OWAAutoLink">https://gitlab.com/petsc/petsc/-/commits/jolivet/feature-matcreatesubmatrices-rectangular-sbaij/</a></div>
</div>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">It is ugly code on top of ugly code, so I didn’t try to get it integrated and just used the branch locally, and then moved to some other stuff.</div>
</div>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">I’ll rebase on top of main and try to get it integrated if it could be useful to you (but I’m traveling right now so everything gets done more slowly, sorry).</div>
</div>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif"> </div>
</div>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">Thanks,</div>
</div>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">Pierre</div>
</div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif"><br>
<br>
</div>
<blockquote style="margin-top:5pt; margin-bottom:5pt">
<div>
<div>
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">My matrix is<br>
[A00 A01;<br>
A01^t A11]<br>
so perhaps with MATNEST I can make use of the block-symmetry at least, and then use MATSBAIJ for<span> </span><br>
A00 and A11 if it's possible to combine matrix types which the manual seems to imply.<span> </span><br>
<br>
Kind regards<br>
Carl-Johan<br>
<br>
<br>
<br>
</div>
<blockquote style="margin-top:5pt; margin-bottom:5pt">
<div style="margin:0cm; font-size:11pt; font-family:Calibri,sans-serif">On 26 Aug 2023, at 10:09 PM, Carl-Johan Thore <<a href="mailto:carljohanthore@gmail.com" style="text-decoration: underline; color: blue;" id="OWAb9b72543-8f00-764e-6f16-836e3a7d72b7" class="OWAAutoLink">carljohanthore@gmail.com</a>>
wrote:<br>
<br>
Hi,<br>
<br>
I'm trying to use PCFIELDSPLIT with MATSBAIJ in PETSc 3.19.4.<span> </span><br>
According to the manual "[t]he fieldsplit preconditioner cannot<span> </span><br>
currently be used with the MATBAIJ or MATSBAIJ data formats if the<span> </span><br>
blocksize is larger than 1". Since my blocksize is exactly 1 it would seem that I can use PCFIELDSPLIT. But this fails with "PETSC ERROR: For symmetric format, iscol must equal isrow"<br>
from MatCreateSubMatrix_MPISBAIJ. Tracing backwards one ends up in<span> </span><br>
fieldsplit.c at<br>
<br>
/* extract the A01 and A10 matrices */ ilink = jac->head;<span> </span><br>
PetscCall(ISComplement(ilink->is_col, rstart, rend, &ccis)); if<span> </span><br>
(jac->offdiag_use_amat) { PetscCall(MatCreateSubMatrix(pc->mat,<span> </span><br>
ilink->is, ccis, MAT_INITIAL_MATRIX, &jac->B)); } else {<br>
PetscCall(MatCreateSubMatrix(pc->pmat, ilink->is, ccis,<span> </span><br>
MAT_INITIAL_MATRIX, &jac->B)); }<br>
<br>
This, since my A01 and A10 are not square, seems to explain why iscol is not equal to isrow.<br>
From this I gather that it is in general NOT possible to use<span> </span><br>
PCFIELDSPLIT with MATSBAIJ even with block size 1?<br>
<br>
Kind regards,<br>
Carl-Johan</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>