<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof"><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> > Because of a combination of settings,
our code passes through this line:</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> ></span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> > https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/fieldsplit/fieldsplit.c?ref_type=heads#L692</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> > </span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> > i.e. the matrices associated with each of the sub-KSPs
of a fieldsplit are destroyed and then re-created later.</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> > The thing is that one of these destroyed matrices had
a near nullspace attached, which is lost because the new matrix does > not have it anymore.</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> > </span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> > Is this a bug or are we missing something?</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> </span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> I just want to get a clear picture. You create a PCFIELDSPLIT,
set it up, then pull out the matrices and attach a nullspace before the solve.</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">We need to solve an SNES. We use dmplex
so we have the jacobian allocated before starting the solve.</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">At setup time we </span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"> 1. define the PC of the KSP of the SNES to be fieldsplit</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"> 2. define the fields with ISes</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"> 3. call PCSetup() to create the sub-KSPs</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"> 4. retrieve the matrix attached to the sub-KSP that needs
the near nullspace and attach it to that matrix</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> At a later time, you start another solve with this PC, and
it has the DIFFERENT_NONZERO_PATTERN flag, so it recreates these matrices and loses your attached nullspace.</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">At a later time, in the jacobian evaluation
we populate the global matrix (i.e. not the matrices attached to each sub-KSPs) and then we set dirichlet bcs with MatZeroRowsColumns() on that same global matrix.</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">For some reason, in serial the near nullspace is not lost
but in parallel the call to MatZeroRowsColumns() does change the non-zero structure (even though the manual says it does not) and then the code goes through that line 692 in fieldsplit.c and the near nullspace is lost.</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> First, does the matrix really change?</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">Well, the matrix during setup is not filled in, just allocated.</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">The thing is that if we set MAT_KEEP_NONZERO_PATTERN to true
with MatSetOption() before setting the dirichlet BCs, then the near nullspace is not lost (because the code does not go through line 692 of fieldsplit.c).</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">So there are (at least) two issues:</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"> 1. Code going through line 692 looses the near nullspace
of the matrices attached to the sub-KSPs</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"> 2. The call to MatZeroRowsColumns() changes then non-zero
structure for MPIAIJ but not for SEQAIJ</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">> Second, I had the same problem. I added callbacks to DM
which allow a nullspace to be automatically attached if you extract a certain subfield. Are you using a DM?</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">Yes. Can you give us an example?</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">Regards</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">--</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);">jeremy</span></div>
<div><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="background-color: rgb(255, 255, 255); margin: 0px;"></div>
</body>
</html>