[petsc-users] MatNest with pc_fieldsplit_type multiplicative yields PetscLayoutSetUp error

[Nate Sime]

Initially, I apologise for not being able to provide a minimal working 
example.

I have a system of 4 PDEs defined in 2 spatial dimensions which I have 
discretised by the FEM.

This includes a Taylor-Hood Stokes system, a modified Helmholtz type 
problem, and a nonlinear advection equation.

I assemble each of the blocks of this 4x4 system into Mats and then put 
those into a MatNest.

Additionally I assemble a 4x4 block preconditioner in the same manner, 
using the standard pressure mass matrix for the continuity equation block.

I am using the following simple options:

-ksp_type fgmres
-pc_type fieldsplit
-pc_fieldsplit_type multiplicative

-fieldsplit_0_ksp_type preonly
-fieldsplit_0_pc_type hypre

-fieldsplit_1_ksp_type preonly
-fieldsplit_1_pc_type hypre

-fieldsplit_2_ksp_type preonly
-fieldsplit_2_pc_type hypre

-fieldsplit_3_ksp_type preonly
-fieldsplit_3_pc_type hypre


Solving the system works and scales reasonably well up until the problem 
extends beyond a size of approximately 50e6 DoF with 192 processes 
across 4 nodes.

Beyond that, I run into the following error with PetscLayoutFindOwner 
which I have attached the error message at the end of the email.

Tracing through the PETSc source, I find that the error arises dealing 
with -pc_fieldsplit_type multiplicative.

Changing to -pc_fieldsplit_type additive, I am able to continue my 
scaling experiments beyond 50e6 with no problem so far.

I was hoping someone may have some basic insight into what I may have 
done naively and incorrectly to cause this issue.

Additionally, please let me know what I have inevitably neglected to 
provide for elucidation.

Nate.


[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: PetscLayoutSetUp() must be called first
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.0, unknown
[0]PETSC ERROR: Unknown Name on a  named 
c456-101.stampede2.tacc.utexas.edu by Tue Oct  2 17:11:27 2018
[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: #1 PetscLayoutFindOwner() line 246 in 
petsc/petsc/include/petscis.h
[0]PETSC ERROR: #2 PetscSFSetGraphLayout() line 511 in 
petsc/petsc/src/vec/is/utils/pmap.c
[0]PETSC ERROR: #3 MatNestCreateAggregateL2G_Private() line 1188 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #4 MatSetUp_NestIS_Private() line 1336 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #5 MatNestSetSubMats_Nest() line 1073 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #6 MatNestSetSubMats() line 1130 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #7 MatCreateNest() line 1400 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #8 MatNestGetRow() line 360 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #9 MatNestFindSubMat() line 398 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #10 MatCreateSubMatrix_Nest() line 425 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #11 MatCreateSubMatrix() line 8023 in 
petsc/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #12 PCSetUp_FieldSplit() line 649 in 
petsc/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
[0]PETSC ERROR: #13 PCSetUp() line 932 in 
petsc/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #14 KSPSetUp() line 391 in 
petsc/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #15 KSPSolve() line 723 in 
petsc/petsc/src/ksp/ksp/interface/itfunc.c