<div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Tue, Oct 2, 2018 at 6:47 PM Nate Sime <<a href="mailto:nsime@carnegiescience.edu">nsime@carnegiescience.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Initially, I apologise for not being able to provide a minimal working <br>
example.<br>
<br>
I have a system of 4 PDEs defined in 2 spatial dimensions which I have <br>
discretised by the FEM.<br>
<br>
This includes a Taylor-Hood Stokes system, a modified Helmholtz type <br>
problem, and a nonlinear advection equation.<br></blockquote><div><br></div><div>This sounds like magma dynamics, but that should have linear advection.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I assemble each of the blocks of this 4x4 system into Mats and then put <br>
those into a MatNest.<br></blockquote><div><br></div><div>We can track down this error. However, I want to first say that I don't think</div><div>MatNest should ever be in your user code. MatNest is only an optimization</div><div>of the storage format, like SBAIJ. It does not contribute anything to functionality.</div><div>If you want the ability to insert into a submatrix, you can use</div><div><br></div><div> <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html</a></div><div><br></div><div>or</div><div><br></div><div> <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateLocalRef.html#MatCreateLocalRef">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateLocalRef.html#MatCreateLocalRef</a></div><div><br></div><div>Then you can start off using just AIJ for testing. I doubt the MAtNest optimization would</div><div>make much of a difference here, but testing it after everything works is much easier.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Additionally I assemble a 4x4 block preconditioner in the same manner, <br>
using the standard pressure mass matrix for the continuity equation block.<br>
<br>
I am using the following simple options:<br>
<br>
-ksp_type fgmres<br>
-pc_type fieldsplit<br>
-pc_fieldsplit_type multiplicative<br>
<br>
-fieldsplit_0_ksp_type preonly<br>
-fieldsplit_0_pc_type hypre<br>
<br>
-fieldsplit_1_ksp_type preonly<br>
-fieldsplit_1_pc_type hypre<br>
<br>
-fieldsplit_2_ksp_type preonly<br>
-fieldsplit_2_pc_type hypre<br>
<br>
-fieldsplit_3_ksp_type preonly<br>
-fieldsplit_3_pc_type hypre<br>
<br>
<br>
Solving the system works and scales reasonably well up until the problem <br>
extends beyond a size of approximately 50e6 DoF with 192 processes <br>
across 4 nodes.<br>
<br>
Beyond that, I run into the following error with PetscLayoutFindOwner <br>
which I have attached the error message at the end of the email.<br>
<br>
Tracing through the PETSc source, I find that the error arises dealing <br>
with -pc_fieldsplit_type multiplicative.<br>
<br>
Changing to -pc_fieldsplit_type additive, I am able to continue my <br>
scaling experiments beyond 50e6 with no problem so far.<br>
<br>
I was hoping someone may have some basic insight into what I may have <br>
done naively and incorrectly to cause this issue.<br>
<br>
Additionally, please let me know what I have inevitably neglected to <br>
provide for elucidation.<br>
<br>
Nate.<br>
<br>
<br>
[0]PETSC ERROR: --------------------- Error Message <br>
--------------------------------------------------------------<br>
[0]PETSC ERROR: Object is in wrong state<br>
[0]PETSC ERROR: PetscLayoutSetUp() must be called first<br>
[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> <br>
for trouble shooting.<br>
[0]PETSC ERROR: Petsc Release Version 3.10.0, unknown<br>
[0]PETSC ERROR: Unknown Name on a named <br>
<a href="http://c456-101.stampede2.tacc.utexas.edu" rel="noreferrer" target="_blank">c456-101.stampede2.tacc.utexas.edu</a> by Tue Oct 2 17:11:27 2018<br>
[0]PETSC ERROR: --------------------- Error Message <br>
--------------------------------------------------------------<br>
[0]PETSC ERROR: #1 PetscLayoutFindOwner() line 246 in <br>
petsc/petsc/include/petscis.h<br>
[0]PETSC ERROR: #2 PetscSFSetGraphLayout() line 511 in <br>
petsc/petsc/src/vec/is/utils/pmap.c<br>
[0]PETSC ERROR: #3 MatNestCreateAggregateL2G_Private() line 1188 in <br>
petsc/petsc/src/mat/impls/nest/matnest.c<br>
[0]PETSC ERROR: #4 MatSetUp_NestIS_Private() line 1336 in <br>
petsc/petsc/src/mat/impls/nest/matnest.c<br>
[0]PETSC ERROR: #5 MatNestSetSubMats_Nest() line 1073 in <br>
petsc/petsc/src/mat/impls/nest/matnest.c<br>
[0]PETSC ERROR: #6 MatNestSetSubMats() line 1130 in <br>
petsc/petsc/src/mat/impls/nest/matnest.c<br>
[0]PETSC ERROR: #7 MatCreateNest() line 1400 in <br>
petsc/petsc/src/mat/impls/nest/matnest.c<br>
[0]PETSC ERROR: #8 MatNestGetRow() line 360 in <br>
petsc/petsc/src/mat/impls/nest/matnest.c<br>
[0]PETSC ERROR: #9 MatNestFindSubMat() line 398 in <br>
petsc/petsc/src/mat/impls/nest/matnest.c<br>
[0]PETSC ERROR: #10 MatCreateSubMatrix_Nest() line 425 in <br>
petsc/petsc/src/mat/impls/nest/matnest.c<br>
[0]PETSC ERROR: #11 MatCreateSubMatrix() line 8023 in <br>
petsc/petsc/src/mat/interface/matrix.c<br>
[0]PETSC ERROR: #12 PCSetUp_FieldSplit() line 649 in <br>
petsc/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c<br>
[0]PETSC ERROR: #13 PCSetUp() line 932 in <br>
petsc/petsc/src/ksp/pc/interface/precon.c<br>
[0]PETSC ERROR: #14 KSPSetUp() line 391 in <br>
petsc/petsc/src/ksp/ksp/interface/itfunc.c<br>
[0]PETSC ERROR: #15 KSPSolve() line 723 in <br>
petsc/petsc/src/ksp/ksp/interface/itfunc.c<br>
<br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div></div></div>