[petsc-users] Diagnosing Convergence Issue in Fieldsplit Problem
Barry Smith
bsmith at petsc.dev
Thu May 30 16:25:42 CDT 2024
Preconditioner for the Schur complement formed from A11
But
PC Object: (fieldsplit_pressure_) 1 MPI process
>
> type: ilu
> out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 2.22045e-14
> matrix ordering: natural
> factor fill ratio given 1., needed 1.
> Factored matrix follows:
> Mat Object: (fieldsplit_pressure_) 1 MPI process
> type: seqaij
> rows=25, cols=25
> package used to perform factorization: petsc
> total: nonzeros=105, allocated nonzeros=105
> not using I-node routines
So it is trying to do ILU on A11 (to use as the preconditioner of the Schur complement). But A11 is identically zero so ILU will produce a zero pivot
So you need to set a different preconditioner for S. This can be done with
-pc_fieldsplit_schur_precondition <self,selfp,user,a11,full> See PCFieldSplitSetSchurPre
First use
-pc_fieldsplit_schur_precondition full -fieldsplit_pressure_pc_type svd (note the full S is singular so LU should fail).
and that should make the PCFIELDSPLIT a direct solver. You then explore cheaper options (since computing S explicitly to produce a preconditioner
is not reasonable except for small problems). So try next
-pc_fieldsplit_schur_precondition self -fieldsplit_pressure_pc_type jacobi
For more advanced tuning Matt can help you out once you have the basics working.
> On May 30, 2024, at 4:49 PM, Colton Bryant <coltonbryant2021 at u.northwestern.edu> wrote:
>
> Hi Barry,
>
> Yes, each index set has the correct entries when I checked manually on a small example.
>
> For the nullspace I was trying to manually build the constant basis on a compatible DMStag containing just the pressure nodes and creating the nullspace from that but that does not seem to work. Using MatNullSpaceCreate(comm,PETSC_TRUE,0,NULL,&sp) does show that the Schur system has the attached null space. However the system still fails to converge with the schur block giving the error
>
> Linear fieldsplit_pressure_ solve did not converge due to DIVERGED_PC_FAILED iterations 0
> PC failed due to FACTOR_NUMERIC_ZEROPIVOT
>
> The output from -ksp_view is
>
> KSP Object: 1 MPI process
> type: fgmres
> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> happy breakdown tolerance 1e-30
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
> right preconditioning
> using UNPRECONDITIONED norm type for convergence test
> PC Object: 1 MPI process
> type: fieldsplit
> FieldSplit with Schur preconditioner, factorization FULL
> Preconditioner for the Schur complement formed from A11
> Split info:
> Split number 0 Defined by IS
> Split number 1 Defined by IS
> KSP solver for A00 block
> KSP Object: (fieldsplit_velocity_) 1 MPI process
> type: gmres
> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> happy breakdown tolerance 1e-30
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
> left preconditioning
> using PRECONDITIONED norm type for convergence test
> PC Object: (fieldsplit_velocity_) 1 MPI process
> type: lu
> out-of-place factorization
> tolerance for zero pivot 2.22045e-14
> matrix ordering: nd
> factor fill ratio given 5., needed 2.17695
> Factored matrix follows:
> Mat Object: (fieldsplit_velocity_) 1 MPI process
> type: seqaij
> rows=60, cols=60
> package used to perform factorization: petsc
> total: nonzeros=1058, allocated nonzeros=1058
> using I-node routines: found 35 nodes, limit used is 5
> linear system matrix = precond matrix:
> Mat Object: (fieldsplit_velocity_) 1 MPI process
> type: seqaij
> rows=60, cols=60
> total: nonzeros=486, allocated nonzeros=486
> total number of mallocs used during MatSetValues calls=0
> using I-node routines: found 35 nodes, limit used is 5
> KSP solver for S = A11 - A10 inv(A00) A01
> KSP Object: (fieldsplit_pressure_) 1 MPI process
> type: gmres
> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> happy breakdown tolerance 1e-30
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
> left preconditioning
> using PRECONDITIONED norm type for convergence test
> PC Object: (fieldsplit_pressure_) 1 MPI process
> type: ilu
> out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 2.22045e-14
> matrix ordering: natural
> factor fill ratio given 1., needed 1.
> Factored matrix follows:
> Mat Object: (fieldsplit_pressure_) 1 MPI process
> type: seqaij
> rows=25, cols=25
> package used to perform factorization: petsc
> total: nonzeros=105, allocated nonzeros=105
> not using I-node routines
> linear system matrix followed by preconditioner matrix:
> Mat Object: (fieldsplit_pressure_) 1 MPI process
> type: schurcomplement
> rows=25, cols=25
> has attached null space
> Schur complement A11 - A10 inv(A00) A01
> A11
> Mat Object: (fieldsplit_pressure_) 1 MPI process
> type: seqaij
> rows=25, cols=25
> total: nonzeros=105, allocated nonzeros=105
> total number of mallocs used during MatSetValues calls=0
> has attached null space
> not using I-node routines
> A10
> Mat Object: 1 MPI process
> type: seqaij
> rows=25, cols=60
> total: nonzeros=220, allocated nonzeros=220
> total number of mallocs used during MatSetValues calls=0
> not using I-node routines
> KSP solver for A00 block viewable with the additional option -fieldsplit_velocity_ksp_view
> A01
> Mat Object: 1 MPI process
> type: seqaij
> rows=60, cols=25
> total: nonzeros=220, allocated nonzeros=220
> total number of mallocs used during MatSetValues calls=0
> using I-node routines: found 35 nodes, limit used is 5
> Mat Object: (fieldsplit_pressure_) 1 MPI process
> type: seqaij
> rows=25, cols=25
> total: nonzeros=105, allocated nonzeros=105
> total number of mallocs used during MatSetValues calls=0
> has attached null space
> not using I-node routines
> linear system matrix = precond matrix:
> Mat Object: 1 MPI process
> type: seqaij
> rows=85, cols=85
> total: nonzeros=1031, allocated nonzeros=1031
> total number of mallocs used during MatSetValues calls=0
> has attached null space
> using I-node routines: found 35 nodes, limit used is 5
>
>
> On Thu, May 30, 2024 at 1:53 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>>
>>
>>> On May 30, 2024, at 3:15 PM, Colton Bryant <coltonbryant2021 at u.northwestern.edu <mailto:coltonbryant2021 at u.northwestern.edu>> wrote:
>>>
>>> Hi Barry,
>>>
>>> Do you know of an example that demonstrates this approach? I have tried implementing this using DMStagCreateISFromStencils and then calling PCFieldSplitSetIS with fields named "velocity" and "pressure" respectively, but when I look at -ksp_view the fields are being set to "fieldsplit_face" and "fieldsplit_element" and as problems are not converging I expect the constant null space is not being attached.
>>
>> First confirm that each IS has the entries you expect
>>
>> Then for the pressure IS are you using PetscObjectCompose((PetscObject*)is,"nullspace", (PetscObject *)sp); where sp is the null space of the pressure variables
>> which I think you can create using MatNullSpaceCreate(comm,PETSC_TRUE,0,NULL,&sp);
>>
>> PCFIELDSPLIT is suppose to snag this null space that you provided and use it on the Shur system. If you run with -ksp_view it should list what matrices have an attached null space.
>>
>>
>>
>>>
>>> Thanks,
>>> Colton
>>>
>>> On Thu, May 23, 2024 at 12:55 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>>>>
>>>> Unfortunately it cannot automatically because -pc_fieldsplit_detect_saddle_point just grabs part of the matrix (having no concept of "what part" so doesn't know to grab the null space information.
>>>>
>>>> It would be possible for PCFIELDSPLIT to access the null space of the larger matrix directly as vectors and check if they are all zero in the 00 block, then it would know that the null space only applied to the second block and could use it for the Schur complement.
>>>>
>>>> Matt, Jed, Stefano, Pierre does this make sense?
>>>>
>>>> Colton,
>>>>
>>>> Meanwhile the quickest thing you can do is to generate the IS the defines the first and second block (instead of using -pc_fieldsplit_detect_saddle_point) and use PetscObjectCompose to attach the constant null space to the second block with the name "nullspace". PCFIELDSPLIT will then use this null space for the Schur complement solve.
>>>>
>>>> Barry
>>>>
>>>>
>>>>> On May 23, 2024, at 2:43 PM, Colton Bryant <coltonbryant2021 at u.northwestern.edu <mailto:coltonbryant2021 at u.northwestern.edu>> wrote:
>>>>>
>>>>> Yes, the original operator definitely has a constant null space corresponding to the constant pressure mode. I am currently handling this by using the MatSetNullSpace function when the matrix is being created. Does this information get passed to the submatrices of the fieldsplit?
>>>>>
>>>>> -Colton
>>>>>
>>>>> On Thu, May 23, 2024 at 12:36 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>>>>>>
>>>>>> Ok,
>>>>>>
>>>>>> So what is happening is that GMRES with a restart of 30 is running on the Schur complement system with no preconditioning and LU (as a direct solver) is being used in the application of S (the Schur complement). The convergence of GMRES is stagnating after getting about 8 digits of accuracy in the residual. Then at the second GMRES
>>>>>> restart it is comparing the explicitly computing residual b - Ax with that computed inside the GMRES algorithm (via a recursive formula) and finding a large difference so generating an error. Since you are using a direct solver on the A_{00} block and it is well-conditioned this problem is not expected.
>>>>>>
>>>>>> Is it possible that the S operator has a null space (perhaps of the constant vector)? Or, relatedly, does your original full matrix have a null space?
>>>>>>
>>>>>> We have a way to associated null spaces of the submatrices in PCFIELDSPLIT by attaching them to the IS that define the fields, but unfortunately not trivially when using -pc_fieldsplit_detect_saddle_point. And sadly the current support seems completely undocumented.
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On May 23, 2024, at 2:16 PM, Colton Bryant <coltonbryant2021 at u.northwestern.edu <mailto:coltonbryant2021 at u.northwestern.edu>> wrote:
>>>>>>>
>>>>>>> Hi Barry,
>>>>>>>
>>>>>>> I saw that was reporting as an unused option and the error message I sent was run with -fieldsplit_0_ksp_type preonly.
>>>>>>>
>>>>>>> -Colton
>>>>>>>
>>>>>>> On Thu, May 23, 2024 at 12:13 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Sorry I gave the wrong option. Use -fieldsplit_0_ksp_type preonly
>>>>>>>>
>>>>>>>> Barry
>>>>>>>>
>>>>>>>>> On May 23, 2024, at 12:51 PM, Colton Bryant <coltonbryant2021 at u.northwestern.edu <mailto:coltonbryant2021 at u.northwestern.edu>> wrote:
>>>>>>>>>
>>>>>>>>> That produces the error:
>>>>>>>>>
>>>>>>>>> [0]PETSC ERROR: Residual norm computed by GMRES recursion formula 2.68054e-07 is far from the computed residual norm 6.86309e-06 at restart, residual norm at start of cycle 2.68804e-07
>>>>>>>>>
>>>>>>>>> The rest of the error is identical.
>>>>>>>>>
>>>>>>>>> On Thu, May 23, 2024 at 10:46 AM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>>>>>>>>>>
>>>>>>>>>> Use -pc_fieldsplit_0_ksp_type preonly
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On May 23, 2024, at 12:43 PM, Colton Bryant <coltonbryant2021 at u.northwestern.edu <mailto:coltonbryant2021 at u.northwestern.edu>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> That produces the following error:
>>>>>>>>>>>
>>>>>>>>>>> [0]PETSC ERROR: Residual norm computed by GMRES recursion formula 2.79175e-07 is far from the computed residual norm 0.000113154 at restart, residual norm at start of cycle 2.83065e-07
>>>>>>>>>>> [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZsCPHC5g2pfiwFqZ9iGAXJb3Jd0OvtH31f70si-azCrYjFOFCmjN6rD5qICGN9V_Ll2kn3o1fkako4PHyz78-ho$ for trouble shooting.
>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.21.0, unknown
>>>>>>>>>>> [0]PETSC ERROR: ./mainOversetLS_exe on a arch-linux-c-opt named glass by colton Thu May 23 10:41:09 2024
>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich --with-cc=gcc --with-cxx=g++ --with-debugging=no --with-fc=gfortran COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 PETSC_ARCH=arch-linux-c-opt --download-sowing
>>>>>>>>>>> [0]PETSC ERROR: #1 KSPGMRESCycle() at /home/colton/petsc/src/ksp/ksp/impls/gmres/gmres.c:115
>>>>>>>>>>> [0]PETSC ERROR: #2 KSPSolve_GMRES() at /home/colton/petsc/src/ksp/ksp/impls/gmres/gmres.c:227
>>>>>>>>>>> [0]PETSC ERROR: #3 KSPSolve_Private() at /home/colton/petsc/src/ksp/ksp/interface/itfunc.c:905
>>>>>>>>>>> [0]PETSC ERROR: #4 KSPSolve() at /home/colton/petsc/src/ksp/ksp/interface/itfunc.c:1078
>>>>>>>>>>> [0]PETSC ERROR: #5 PCApply_FieldSplit_Schur() at /home/colton/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1203
>>>>>>>>>>> [0]PETSC ERROR: #6 PCApply() at /home/colton/petsc/src/ksp/pc/interface/precon.c:497
>>>>>>>>>>> [0]PETSC ERROR: #7 KSP_PCApply() at /home/colton/petsc/include/petsc/private/kspimpl.h:409
>>>>>>>>>>> [0]PETSC ERROR: #8 KSPFGMRESCycle() at /home/colton/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:123
>>>>>>>>>>> [0]PETSC ERROR: #9 KSPSolve_FGMRES() at /home/colton/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:235
>>>>>>>>>>> [0]PETSC ERROR: #10 KSPSolve_Private() at /home/colton/petsc/src/ksp/ksp/interface/itfunc.c:905
>>>>>>>>>>> [0]PETSC ERROR: #11 KSPSolve() at /home/colton/petsc/src/ksp/ksp/interface/itfunc.c:1078
>>>>>>>>>>> [0]PETSC ERROR: #12 solveStokes() at cartesianStokesGrid.cpp:1403
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, May 23, 2024 at 10:33 AM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Run the failing case with also -ksp_error_if_not_converged so we see exactly where the problem is first detected.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On May 23, 2024, at 11:51 AM, Colton Bryant <coltonbryant2021 at u.northwestern.edu <mailto:coltonbryant2021 at u.northwestern.edu>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Barry,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for letting me know about the need to use fgmres in this case. I ran a smaller problem (1230 in the first block) and saw similar behavior in the true residual.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I also ran the same problem with the options -fieldsplit_0_pc_type svd -fieldsplit_0_pc_svd_monitor and get the following output:
>>>>>>>>>>>>> SVD: condition number 1.933639985881e+03, 0 of 1230 singular values are (nearly) zero
>>>>>>>>>>>>> SVD: smallest singular values: 4.132036392141e-03 4.166444542385e-03 4.669534028645e-03 4.845532162256e-03 5.047038625390e-03
>>>>>>>>>>>>> SVD: largest singular values : 7.947990616611e+00 7.961437414477e+00 7.961851612473e+00 7.971335373142e+00 7.989870790960e+00
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would be surprised if the A_{00} block is ill conditioned as it's just a standard discretization of the laplacian with some rows replaced with ones on the diagonal due to interpolations from the overset mesh. I'm wondering if I'm somehow violating a solvability condition of the problem?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the help!
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Colton
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, May 22, 2024 at 6:09 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the info. I see you are using GMRES inside the Schur complement solver, this is ok but when you do you need to use fgmres as the outer solver. But this is unlikely to be the cause of the exact problem you are seeing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not sure why the Schur complement KSP is suddenly seeing a large increase in the true residual norm. Is it possible the A_{00} block is ill-conditioned?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you run with a smaller problem? Say 2,000 or so in the first block? Is there still a problem?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On May 22, 2024, at 6:00 PM, Colton Bryant <coltonbryant2021 at u.northwestern.edu <mailto:coltonbryant2021 at u.northwestern.edu>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Barry,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have not used any other solver parameters in the code and the full set of solver related command line options are those I mentioned in the previous email.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Below is the output from -ksp_view:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> KSP Object: (back_) 1 MPI process
>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>> happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000.
>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>> PC Object: (back_) 1 MPI process
>>>>>>>>>>>>>>> type: fieldsplit
>>>>>>>>>>>>>>> FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL
>>>>>>>>>>>>>>> Preconditioner for the Schur complement formed from S itself
>>>>>>>>>>>>>>> Split info:
>>>>>>>>>>>>>>> Split number 0 Defined by IS
>>>>>>>>>>>>>>> Split number 1 Defined by IS
>>>>>>>>>>>>>>> KSP solver for A00 block
>>>>>>>>>>>>>>> KSP Object: (back_fieldsplit_0_) 1 MPI process
>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>> happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>> PC Object: (back_fieldsplit_0_) 1 MPI process
>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>> out-of-place factorization
>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>> factor fill ratio given 5., needed 8.83482
>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>> Mat Object: (back_fieldsplit_0_) 1 MPI process
>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>> rows=30150, cols=30150
>>>>>>>>>>>>>>> package used to perform factorization: petsc
>>>>>>>>>>>>>>> total: nonzeros=2649120, allocated nonzeros=2649120
>>>>>>>>>>>>>>> using I-node routines: found 15019 nodes, limit used is 5
>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>> Mat Object: (back_fieldsplit_0_) 1 MPI process
>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>> rows=30150, cols=30150
>>>>>>>>>>>>>>> total: nonzeros=299850, allocated nonzeros=299850
>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls=0
>>>>>>>>>>>>>>> using I-node routines: found 15150 nodes, limit used is 5
>>>>>>>>>>>>>>> KSP solver for S = A11 - A10 inv(A00) A01
>>>>>>>>>>>>>>> KSP Object: (back_fieldsplit_1_) 1 MPI process
>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>> happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000.
>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>> PC Object: (back_fieldsplit_1_) 1 MPI process
>>>>>>>>>>>>>>> type: none
>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>> Mat Object: (back_fieldsplit_1_) 1 MPI process
>>>>>>>>>>>>>>> type: schurcomplement
>>>>>>>>>>>>>>> rows=15000, cols=15000
>>>>>>>>>>>>>>> Schur complement A11 - A10 inv(A00) A01
>>>>>>>>>>>>>>> A11
>>>>>>>>>>>>>>> Mat Object: (back_fieldsplit_1_) 1 MPI process
>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>> rows=15000, cols=15000
>>>>>>>>>>>>>>> total: nonzeros=74700, allocated nonzeros=74700
>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls=0
>>>>>>>>>>>>>>> not using I-node routines
>>>>>>>>>>>>>>> A10
>>>>>>>>>>>>>>> Mat Object: 1 MPI process
>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>> rows=15000, cols=30150
>>>>>>>>>>>>>>> total: nonzeros=149550, allocated nonzeros=149550
>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls=0
>>>>>>>>>>>>>>> not using I-node routines
>>>>>>>>>>>>>>> KSP solver for A00 block viewable with the additional option -back_fieldsplit_0_ksp_view
>>>>>>>>>>>>>>> A01
>>>>>>>>>>>>>>> Mat Object: 1 MPI process
>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>> rows=30150, cols=15000
>>>>>>>>>>>>>>> total: nonzeros=149550, allocated nonzeros=149550
>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls=0
>>>>>>>>>>>>>>> using I-node routines: found 15150 nodes, limit used is 5
>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>> Mat Object: (back_) 1 MPI process
>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>> rows=45150, cols=45150
>>>>>>>>>>>>>>> total: nonzeros=673650, allocated nonzeros=673650
>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls=0
>>>>>>>>>>>>>>> has attached null space
>>>>>>>>>>>>>>> using I-node routines: found 15150 nodes, limit used is 5
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks again!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Colton
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, May 22, 2024 at 3:39 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Are you using any other command line options or did you hardwire any solver parameters in the code with, like, KSPSetXXX() or PCSetXXX() Please send all of them.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Something funky definitely happened when the true residual norms jumped up.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Could you run the same thing with -ksp_view and don't use any thing like -ksp_error_if_not_converged so we can see exactly what is being run.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On May 22, 2024, at 3:21 PM, Colton Bryant <coltonbryant2021 at u.northwestern.edu <mailto:coltonbryant2021 at u.northwestern.edu>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This Message Is From an External Sender
>>>>>>>>>>>>>>>>> This message came from outside your organization.
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am solving the Stokes equations on a MAC grid discretized by finite differences using a DMSTAG object. I have tested the solver quite extensively on manufactured problems and it seems to work well. As I am still just trying to get things working and not yet worried about speed I am using the following solver options:
>>>>>>>>>>>>>>>>> -pc_type fieldsplit
>>>>>>>>>>>>>>>>> -pc_fieldsplit_detect_saddle_point
>>>>>>>>>>>>>>>>> -fieldsplit_0_pc_type lu
>>>>>>>>>>>>>>>>> -fieldsplit_1_ksp_rtol 1.e-8
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However I am now using this solver as an inner step of a larger code and have run into issues. The code repeatedly solves the Stokes equations with varying right hand sides coming from changing problem geometry (the solver is a part of an overset grid scheme coupled to a level set method evolving in time). After a couple timesteps I observe the following output when running with -fieldsplit_1_ksp_converged_reason -fieldsplit_1_ksp_monitor_true_residual:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Residual norms for back_fieldsplit_1_ solve.
>>>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.826514299465e-02 true resid norm 2.826514299465e-02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 7.286621865915e-03 true resid norm 7.286621865915e-03 ||r(i)||/||b|| 2.577953300039e-01
>>>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.500598474492e-03 true resid norm 1.500598474492e-03 ||r(i)||/||b|| 5.309007192273e-02
>>>>>>>>>>>>>>>>> 3 KSP preconditioned resid norm 3.796396924978e-04 true resid norm 3.796396924978e-04 ||r(i)||/||b|| 1.343137349666e-02
>>>>>>>>>>>>>>>>> 4 KSP preconditioned resid norm 8.091057439816e-05 true resid norm 8.091057439816e-05 ||r(i)||/||b|| 2.862556697960e-03
>>>>>>>>>>>>>>>>> 5 KSP preconditioned resid norm 3.689113122359e-05 true resid norm 3.689113122359e-05 ||r(i)||/||b|| 1.305181128239e-03
>>>>>>>>>>>>>>>>> 6 KSP preconditioned resid norm 2.116450533352e-05 true resid norm 2.116450533352e-05 ||r(i)||/||b|| 7.487846545662e-04
>>>>>>>>>>>>>>>>> 7 KSP preconditioned resid norm 3.968234031201e-06 true resid norm 3.968234031200e-06 ||r(i)||/||b|| 1.403932055801e-04
>>>>>>>>>>>>>>>>> 8 KSP preconditioned resid norm 6.666949419511e-07 true resid norm 6.666949419506e-07 ||r(i)||/||b|| 2.358717739644e-05
>>>>>>>>>>>>>>>>> 9 KSP preconditioned resid norm 1.941522884928e-07 true resid norm 1.941522884931e-07 ||r(i)||/||b|| 6.868965372998e-06
>>>>>>>>>>>>>>>>> 10 KSP preconditioned resid norm 6.729545258682e-08 true resid norm 6.729545258626e-08 ||r(i)||/||b|| 2.380863687793e-06
>>>>>>>>>>>>>>>>> 11 KSP preconditioned resid norm 3.009070131709e-08 true resid norm 3.009070131735e-08 ||r(i)||/||b|| 1.064586912687e-06
>>>>>>>>>>>>>>>>> 12 KSP preconditioned resid norm 7.849353009588e-09 true resid norm 7.849353009903e-09 ||r(i)||/||b|| 2.777043445840e-07
>>>>>>>>>>>>>>>>> 13 KSP preconditioned resid norm 2.306283345754e-09 true resid norm 2.306283346677e-09 ||r(i)||/||b|| 8.159461097060e-08
>>>>>>>>>>>>>>>>> 14 KSP preconditioned resid norm 9.336302495083e-10 true resid norm 9.336302502503e-10 ||r(i)||/||b|| 3.303115255517e-08
>>>>>>>>>>>>>>>>> 15 KSP preconditioned resid norm 6.537456143401e-10 true resid norm 6.537456141617e-10 ||r(i)||/||b|| 2.312903968982e-08
>>>>>>>>>>>>>>>>> 16 KSP preconditioned resid norm 6.389159552788e-10 true resid norm 6.389159550304e-10 ||r(i)||/||b|| 2.260437724130e-08
>>>>>>>>>>>>>>>>> 17 KSP preconditioned resid norm 6.380905134246e-10 true resid norm 6.380905136023e-10 ||r(i)||/||b|| 2.257517372981e-08
>>>>>>>>>>>>>>>>> 18 KSP preconditioned resid norm 6.380440605992e-10 true resid norm 6.380440604688e-10 ||r(i)||/||b|| 2.257353025207e-08
>>>>>>>>>>>>>>>>> 19 KSP preconditioned resid norm 6.380427156582e-10 true resid norm 6.380427157894e-10 ||r(i)||/||b|| 2.257348267830e-08
>>>>>>>>>>>>>>>>> 20 KSP preconditioned resid norm 6.380426714897e-10 true resid norm 6.380426714004e-10 ||r(i)||/||b|| 2.257348110785e-08
>>>>>>>>>>>>>>>>> 21 KSP preconditioned resid norm 6.380426656970e-10 true resid norm 6.380426658839e-10 ||r(i)||/||b|| 2.257348091268e-08
>>>>>>>>>>>>>>>>> 22 KSP preconditioned resid norm 6.380426650538e-10 true resid norm 6.380426650287e-10 ||r(i)||/||b|| 2.257348088242e-08
>>>>>>>>>>>>>>>>> 23 KSP preconditioned resid norm 6.380426649918e-10 true resid norm 6.380426645888e-10 ||r(i)||/||b|| 2.257348086686e-08
>>>>>>>>>>>>>>>>> 24 KSP preconditioned resid norm 6.380426649803e-10 true resid norm 6.380426644294e-10 ||r(i)||/||b|| 2.257348086122e-08
>>>>>>>>>>>>>>>>> 25 KSP preconditioned resid norm 6.380426649796e-10 true resid norm 6.380426649774e-10 ||r(i)||/||b|| 2.257348088061e-08
>>>>>>>>>>>>>>>>> 26 KSP preconditioned resid norm 6.380426649795e-10 true resid norm 6.380426653788e-10 ||r(i)||/||b|| 2.257348089481e-08
>>>>>>>>>>>>>>>>> 27 KSP preconditioned resid norm 6.380426649795e-10 true resid norm 6.380426646744e-10 ||r(i)||/||b|| 2.257348086989e-08
>>>>>>>>>>>>>>>>> 28 KSP preconditioned resid norm 6.380426649795e-10 true resid norm 6.380426650818e-10 ||r(i)||/||b|| 2.257348088430e-08
>>>>>>>>>>>>>>>>> 29 KSP preconditioned resid norm 6.380426649795e-10 true resid norm 6.380426649518e-10 ||r(i)||/||b|| 2.257348087970e-08
>>>>>>>>>>>>>>>>> 30 KSP preconditioned resid norm 6.380426652142e-10 true resid norm 6.380426652142e-10 ||r(i)||/||b|| 2.257348088898e-08
>>>>>>>>>>>>>>>>> 31 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426646799e-10 ||r(i)||/||b|| 2.257348087008e-08
>>>>>>>>>>>>>>>>> 32 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426648077e-10 ||r(i)||/||b|| 2.257348087460e-08
>>>>>>>>>>>>>>>>> 33 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426649048e-10 ||r(i)||/||b|| 2.257348087804e-08
>>>>>>>>>>>>>>>>> 34 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426648142e-10 ||r(i)||/||b|| 2.257348087483e-08
>>>>>>>>>>>>>>>>> 35 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651079e-10 ||r(i)||/||b|| 2.257348088522e-08
>>>>>>>>>>>>>>>>> 36 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650433e-10 ||r(i)||/||b|| 2.257348088294e-08
>>>>>>>>>>>>>>>>> 37 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426649765e-10 ||r(i)||/||b|| 2.257348088057e-08
>>>>>>>>>>>>>>>>> 38 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650364e-10 ||r(i)||/||b|| 2.257348088269e-08
>>>>>>>>>>>>>>>>> 39 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650051e-10 ||r(i)||/||b|| 2.257348088159e-08
>>>>>>>>>>>>>>>>> 40 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651154e-10 ||r(i)||/||b|| 2.257348088549e-08
>>>>>>>>>>>>>>>>> 41 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650246e-10 ||r(i)||/||b|| 2.257348088227e-08
>>>>>>>>>>>>>>>>> 42 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650702e-10 ||r(i)||/||b|| 2.257348088389e-08
>>>>>>>>>>>>>>>>> 43 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651686e-10 ||r(i)||/||b|| 2.257348088737e-08
>>>>>>>>>>>>>>>>> 44 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650870e-10 ||r(i)||/||b|| 2.257348088448e-08
>>>>>>>>>>>>>>>>> 45 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651208e-10 ||r(i)||/||b|| 2.257348088568e-08
>>>>>>>>>>>>>>>>> 46 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651441e-10 ||r(i)||/||b|| 2.257348088650e-08
>>>>>>>>>>>>>>>>> 47 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650955e-10 ||r(i)||/||b|| 2.257348088478e-08
>>>>>>>>>>>>>>>>> 48 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650877e-10 ||r(i)||/||b|| 2.257348088451e-08
>>>>>>>>>>>>>>>>> 49 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651240e-10 ||r(i)||/||b|| 2.257348088579e-08
>>>>>>>>>>>>>>>>> 50 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650534e-10 ||r(i)||/||b|| 2.257348088329e-08
>>>>>>>>>>>>>>>>> 51 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426648615e-10 ||r(i)||/||b|| 2.257348087651e-08
>>>>>>>>>>>>>>>>> 52 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426649523e-10 ||r(i)||/||b|| 2.257348087972e-08
>>>>>>>>>>>>>>>>> 53 KSP preconditioned resid norm 6.380426652140e-10 true resid norm 6.380426652601e-10 ||r(i)||/||b|| 2.257348089061e-08
>>>>>>>>>>>>>>>>> 54 KSP preconditioned resid norm 6.380426652125e-10 true resid norm 6.380427512852e-10 ||r(i)||/||b|| 2.257348393411e-08
>>>>>>>>>>>>>>>>> 55 KSP preconditioned resid norm 6.380426651849e-10 true resid norm 6.380603444402e-10 ||r(i)||/||b|| 2.257410636701e-08
>>>>>>>>>>>>>>>>> 56 KSP preconditioned resid norm 6.380426646751e-10 true resid norm 6.439925413105e-10 ||r(i)||/||b|| 2.278398313542e-08
>>>>>>>>>>>>>>>>> 57 KSP preconditioned resid norm 6.380426514019e-10 true resid norm 2.674218007058e-09 ||r(i)||/||b|| 9.461186902765e-08
>>>>>>>>>>>>>>>>> 58 KSP preconditioned resid norm 6.380425077384e-10 true resid norm 2.406759314486e-08 ||r(i)||/||b|| 8.514937691775e-07
>>>>>>>>>>>>>>>>> 59 KSP preconditioned resid norm 6.380406171326e-10 true resid norm 3.100137288622e-07 ||r(i)||/||b|| 1.096805803957e-05
>>>>>>>>>>>>>>>>> Linear back_fieldsplit_1_ solve did not converge due to DIVERGED_BREAKDOWN iterations 60
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Any advice on steps I could take to elucidate the issue would be greatly appreciated. Thanks so much for any help in advance!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Colton Bryant
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240530/fe4d6537/attachment-0001.html>
More information about the petsc-users
mailing list