[petsc-users] PCFIELDSPLIT with MATSBAIJ

[Carl-Johan Thore]

I've tried the new files, and with them, PCFIELDSPLIT now gets set up
without crashes (but the setup is significantly slower than for MATAIJ)
Unfortunately I still get errors later in the process:

[0]PETSC ERROR: Null argument, when expecting valid pointer
[0]PETSC ERROR: Null Pointer: Parameter # 1
[0]PETSC ERROR: Petsc Development GIT revision: v3.19.4-1023-ga6d78fcba1d
 GIT Date: 2023-08-22 20:32:33 -0400
[0]PETSC ERROR: Configure options -f --with-fortran-bindings=0 --with-cuda
--with-cusp --download-scalapack --download-hdf5 --download-zlib
--download-mumps --download-parmetis --download-metis --download-ptscotch
--download-hypre --download-spai
[0]PETSC ERROR: #1 PetscObjectQuery() at
/mnt/c/mathware/petsc/petsc-v3-19-4/src/sys/objects/inherit.c:742
[0]PETSC ERROR: #2 MatCreateSubMatrix_MPISBAIJ() at
/mnt/c/mathware/petsc/petsc-v3-19-4/src/mat/impls/sbaij/mpi/mpisbaij.c:1414
[0]PETSC ERROR: #3 MatCreateSubMatrix() at
/mnt/c/mathware/petsc/petsc-v3-19-4/src/mat/interface/matrix.c:8476
[0]PETSC ERROR: #4 PCSetUp_FieldSplit() at
/mnt/c/mathware/petsc/petsc-v3-19-4/src/ksp/pc/impls/fieldsplit/fieldsplit.c:826
[0]PETSC ERROR: #5 PCSetUp() at
/mnt/c/mathware/petsc/petsc-v3-19-4/src/ksp/pc/interface/precon.c:1069
[0]PETSC ERROR: #6 KSPSetUp() at
/mnt/c/mathware/petsc/petsc-v3-19-4/src/ksp/ksp/interface/itfunc.c:415

The code I'm running here works without any problems for MATAIJ. To run it
with MATSBAIJ I've simply used the command-line option
-dm_mat_type sbaij


Kind regards,
Carl-Johan


On Sat, Aug 26, 2023 at 5:21 PM Pierre Jolivet via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
>
> On 27 Aug 2023, at 12:14 AM, Carl-Johan Thore <carl-johan.thore at liu.se>
> wrote:
>
> “Well, your A00 and A11 will possibly be SBAIJ also, so you’ll end up with
> the same issue.”
> I’m not sure I follow. Does PCFIELDSPLIT extract further submatrices from
> these blocks, or is there
> somewhere else in the code that things will go wrong?
>
>
> Ah, no, you are right, in that case it should work.
>
> For the MATNEST I was thinking to get some savings from the block-symmetry
> at least
> even if symmetry in A00 and A11 cannot be exploited; using SBAIJ for them
> would just be a
> (pretty big) bonus.
>
> “I’ll rebase on top of main and try to get it integrated if it could be
> useful to you (but I’m traveling
> right now so everything gets done more slowly, sorry).”
> Sound great, Thanks again!
>
>
> The MR is there https://gitlab.com/petsc/petsc/-/merge_requests/6841.
> I need to add a new code path in MatCreateRedundantMatrix() to make sure
> the resulting Mat is indeed SBAIJ, but that is orthogonal to the
> PCFIELDSPLIT issue.
> The branch should be usable in its current state.
>
> Thanks,
> Pierre
>
>
> *From:* Pierre Jolivet <pierre.jolivet at lip6.fr>
> *Sent:* Saturday, August 26, 2023 4:36 PM
> *To:* Carl-Johan Thore <carl-johan.thore at liu.se>
> *Cc:* Carl-Johan Thore <carljohanthore at gmail.com>; petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] PCFIELDSPLIT with MATSBAIJ
>
>
>
>
> On 26 Aug 2023, at 11:16 PM, Carl-Johan Thore <carl-johan.thore at liu.se>
> wrote:
>
> "(Sadly) MATSBAIJ is extremely broken, in particular, it cannot be used to
> retrieve rectangular blocks in MatCreateSubMatrices, thus you cannot get
> the A01 and A10 blocks in PCFIELDSPLIT.
> I have a branch that fixes this, but I haven’t rebased in a while (and I’m
> AFK right now), would you want me to rebase and give it a go, or must you
> stick to a release tarball?"
>
> Ok, would be great if you could look at this! I don't need to stick to any
> particular branch.
>
> Do you think MATNEST could be an alternative here?
>
>
> Well, your A00 and A11 will possibly be SBAIJ also, so you’ll end up with
> the same issue.
> I’m using both approaches (monolithic SBAIJ or Nest + SBAIJ), it was
> crashing but I think it was thoroughly fixed in
> https://gitlab.com/petsc/petsc/-/commits/jolivet/feature-matcreatesubmatrices-rectangular-sbaij/
> It is ugly code on top of ugly code, so I didn’t try to get it integrated
> and just used the branch locally, and then moved to some other stuff.
> I’ll rebase on top of main and try to get it integrated if it could be
> useful to you (but I’m traveling right now so everything gets done more
> slowly, sorry).
>
> Thanks,
> Pierre
>
>
> My matrix is
> [A00 A01;
> A01^t A11]
> so perhaps with MATNEST I can make use of the block-symmetry at least, and
> then use MATSBAIJ for
> A00 and A11 if it's possible to combine matrix types which the manual
> seems to imply.
>
> Kind regards
> Carl-Johan
>
>
>
> On 26 Aug 2023, at 10:09 PM, Carl-Johan Thore <carljohanthore at gmail.com>
> wrote:
> 
> Hi,
>
> I'm trying to use PCFIELDSPLIT with MATSBAIJ in PETSc 3.19.4.
> According to the manual "[t]he fieldsplit preconditioner cannot
> currently be used with the MATBAIJ or MATSBAIJ data formats if the
> blocksize is larger than 1". Since my blocksize is exactly 1 it would seem
> that I can use PCFIELDSPLIT. But this fails with "PETSC ERROR: For
> symmetric format, iscol must equal isrow"
> from MatCreateSubMatrix_MPISBAIJ. Tracing backwards one ends up in
> fieldsplit.c at
>
> /* extract the A01 and A10 matrices */ ilink = jac->head;
> PetscCall(ISComplement(ilink->is_col, rstart, rend, &ccis)); if
> (jac->offdiag_use_amat) { PetscCall(MatCreateSubMatrix(pc->mat,
> ilink->is, ccis, MAT_INITIAL_MATRIX, &jac->B)); } else {
>        PetscCall(MatCreateSubMatrix(pc->pmat, ilink->is, ccis,
> MAT_INITIAL_MATRIX, &jac->B)); }
>
> This, since my A01 and A10 are not square, seems to explain why iscol is
> not equal to isrow.
> From this I gather that it is in general NOT possible to use
> PCFIELDSPLIT with MATSBAIJ even with block size 1?
>
> Kind regards,
> Carl-Johan
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230828/6d37e642/attachment.html>