[petsc-users] MatNest with pc_fieldsplit_type multiplicative yields PetscLayoutSetUp error

[Matthew Knepley]

On Tue, Oct 2, 2018 at 6:47 PM Nate Sime <nsime at carnegiescience.edu> wrote:

> Initially, I apologise for not being able to provide a minimal working
> example.
>
> I have a system of 4 PDEs defined in 2 spatial dimensions which I have
> discretised by the FEM.
>
> This includes a Taylor-Hood Stokes system, a modified Helmholtz type
> problem, and a nonlinear advection equation.
>

This sounds like magma dynamics, but that should have linear advection.


> I assemble each of the blocks of this 4x4 system into Mats and then put
> those into a MatNest.
>

We can track down this error. However, I want to first say that I don't
think
MatNest should ever be in your user code. MatNest is only an optimization
of the storage format, like SBAIJ. It does not contribute anything to
functionality.
If you want the ability to insert into a submatrix, you can use


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html

or


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateLocalRef.html#MatCreateLocalRef

Then you can start off using just AIJ for testing. I doubt the MAtNest
optimization would
make much of a difference here, but testing it after everything works is
much easier.

  Thanks,

     Matt


> Additionally I assemble a 4x4 block preconditioner in the same manner,
> using the standard pressure mass matrix for the continuity equation block.
>
> I am using the following simple options:
>
> -ksp_type fgmres
> -pc_type fieldsplit
> -pc_fieldsplit_type multiplicative
>
> -fieldsplit_0_ksp_type preonly
> -fieldsplit_0_pc_type hypre
>
> -fieldsplit_1_ksp_type preonly
> -fieldsplit_1_pc_type hypre
>
> -fieldsplit_2_ksp_type preonly
> -fieldsplit_2_pc_type hypre
>
> -fieldsplit_3_ksp_type preonly
> -fieldsplit_3_pc_type hypre
>
>
> Solving the system works and scales reasonably well up until the problem
> extends beyond a size of approximately 50e6 DoF with 192 processes
> across 4 nodes.
>
> Beyond that, I run into the following error with PetscLayoutFindOwner
> which I have attached the error message at the end of the email.
>
> Tracing through the PETSc source, I find that the error arises dealing
> with -pc_fieldsplit_type multiplicative.
>
> Changing to -pc_fieldsplit_type additive, I am able to continue my
> scaling experiments beyond 50e6 with no problem so far.
>
> I was hoping someone may have some basic insight into what I may have
> done naively and incorrectly to cause this issue.
>
> Additionally, please let me know what I have inevitably neglected to
> provide for elucidation.
>
> Nate.
>
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: PetscLayoutSetUp() must be called first
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.0, unknown
> [0]PETSC ERROR: Unknown Name on a  named
> c456-101.stampede2.tacc.utexas.edu by Tue Oct  2 17:11:27 2018
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: #1 PetscLayoutFindOwner() line 246 in
> petsc/petsc/include/petscis.h
> [0]PETSC ERROR: #2 PetscSFSetGraphLayout() line 511 in
> petsc/petsc/src/vec/is/utils/pmap.c
> [0]PETSC ERROR: #3 MatNestCreateAggregateL2G_Private() line 1188 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #4 MatSetUp_NestIS_Private() line 1336 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #5 MatNestSetSubMats_Nest() line 1073 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #6 MatNestSetSubMats() line 1130 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #7 MatCreateNest() line 1400 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #8 MatNestGetRow() line 360 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #9 MatNestFindSubMat() line 398 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #10 MatCreateSubMatrix_Nest() line 425 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #11 MatCreateSubMatrix() line 8023 in
> petsc/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #12 PCSetUp_FieldSplit() line 649 in
> petsc/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> [0]PETSC ERROR: #13 PCSetUp() line 932 in
> petsc/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #14 KSPSetUp() line 391 in
> petsc/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #15 KSPSolve() line 723 in
> petsc/petsc/src/ksp/ksp/interface/itfunc.c
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/9bfa48f8/attachment.html>