[petsc-dev] Recover from TS failure

Wed Nov 1 13:45:22 CDT 2017

On Wed, Nov 1, 2017 at 2:34 PM, Jed Brown <jed at jedbrown.org> wrote:

> Mark Adams <mfadams at lbl.gov> writes:
>
> > Oh, Maybe the Jacobian has Nans or Infs even though the last time step
> > survived. Maybe it was going crazy. I'll check
>
> If that is the case you would use TSSetFunctionDomainError().
>

Ah, very cool. Good to know. That does not seem to be the problem. Not sure
what would cause this linear solver error. I'll try different solvers and
see if I get any hints,
Thanks again,

>
> > On Wed, Nov 1, 2017 at 2:13 PM, Mark Adams <mfadams at lbl.gov> wrote:
> >
> >> Yea, I don't understand the linear solve error:
> >>
> >> -ts_monitor -ts_type beuler -pc_type lu -pc_factor_mat_solver_package
> >> mumps -ksp_type preonly -snes_monitor -snes_rtol 1.e-10 -snes_stol
> 1.e-10
> >> -snes_converged_reason -snes_atol 1.e-18 -snes_converged_reason
> >>
> >> Maybe mumps?
> >>
> >> On Wed, Nov 1, 2017 at 2:02 PM, Jed Brown <jed at jedbrown.org> wrote:
> >>
> >>> Mark Adams <mfadams at lbl.gov> writes:
> >>>
> >>> > On Wed, Nov 1, 2017 at 1:46 PM, Jed Brown <jed at jedbrown.org> wrote:
> >>> >
> >>> >> Mark Adams <mfadams at lbl.gov> writes:
> >>> >>
> >>> >> > I have added some code in a TS post step method to look at the
> >>> number of
> >>> >> > nonlinear iterations and cut the time step if it took too many
> SNES
> >>> >> > iterations.  That helped but now I want to go one step further and
> >>> >> recover
> >>> >> > from a failed time step (see appended error message).
> >>> >> >
> >>> >> > How can/should I recover from a SNES failure inside of TS? I could
> >>> just
> >>> >> put
> >>> >> > TSSolve in a loop and if TS failed because of the solver, then
> >>> reduce the
> >>> >> > time step and try to recover the state, and try again.
> >>> >> >
> >>> >> > Thoughts?
> >>> >> >
> >>> >> > Thanks,
> >>> >> > Mark
> >>> >> >
> >>> >> > Note, this is a direct solver and the linear solver is diverging?
> >>> I've
> >>> >> > seen the same thing with the maximum its in SNES and failed
> >>> linesearch.
> >>> >> >
> >>> >> > 23 TS dt 0.0025 time 0.1025
> >>> >> >     0 SNES Function norm 9.997473571601e+04
> >>> >> >     1 SNES Function norm 2.295851692533e-01
> >>> >> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> >> iterations 1
> >>> >> > [0]PETSC ERROR: --------------------- Error Message
> >>> >> > --------------------------------------------------------------
> >>> >> > [0]PETSC ERROR:
> >>> >> > [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE,
> >>> >> increase
> >>> >> > -ts_max_snes_failures or make negative to attempt recovery
> >>> >>
> >>> >> Did you increase -ts_max_snes_failures (TSSetMaxSNESFailures)?
> >>> >>
> >>> >
> >>> > Ah, read the error message :) with a negative value I get this.
> >>> >
> >>> > TS can not, or is not going to try, to reconstruct my state and
> reduce
> >>> the
> >>> > time step. So I think I need to do this myself.
> >>>
> >>> I don't understand.  Why is the linear solve diverging
> >>> (-ksp_converged_reason)?  What TS are you using?
> >>>
> >>> > I guess just making loop and testing in the TSConvergedReason, is the
> >>> way
> >>> > to go. We already do something like this for mesh adpativity.
> >>> >
> >>> > Thanks,
> >>> >
> >>> > 23 TS dt 0.0025 time 0.1025
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >     1 SNES Function norm 2.295851692533e-01
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 1
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 0
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 0
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 0
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 0
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 0
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 0
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 0
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 0
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 0
> >>> >     0 SNES Function norm 9.997473571601e+04
> >>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> >>> iterations 0
> >>> > [0]PETSC ERROR: --------------------- Error Message
> >>> > --------------------------------------------------------------
> >>> > [0]PETSC ERROR:
> >>> > [0]PETSC ERROR: TSStep has failed due to DIVERGED_STEP_REJECTED
> >>> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/
> documentation/faq.html
> >>> for
> >>> > trouble shooting.
> >>> > [0]PETSC ERROR: Petsc Development GIT revision: v3.8-53-ga2d93b2d43
> GIT
> >>> > Date: 2017-10-09 12:56:49 -0400
> >>> > [0]PETSC ERROR: ./ex48 on a arch-macosx-gnu-O named MarksMac-5.local
> by
> >>> > markadams Wed Nov  1 13:51:24 2017
> >>> > [0]PETSC ERROR: Configure options --with-cc=clang --with-cc++=clang++
> >>> > COPTFLAGS="-O2 -g -mavx2" CXXOPTFLAGS="-O2 -g -mavx2" FOPTFLAGS="-O2
> -g
> >>> > -mavx2" --download-mpich=1 --download-parmetis=1 --download-metis=1
> >>> > --download-hypre=1 --download-ml=1 --download-triangle=1
> >>> > --download-ctetgen=1 --download-p4est=1 --download-fftw
> >>> --download-mumps=1
> >>> > --download-scalapack=1 --with-x=0 --download-superlu_dist
> >>> > --download-superlu --download-ctetgen --with-debugging=0
> >>> --download-hdf5=1
> >>> > PETSC_ARCH=arch-macosx-gnu-O --download-chaco
> --with-viewfromoptions=1
> >>> > [0]PETSC ERROR: #1 TSStep() line 4130 in
> >>> > /Users/markadams/Codes/petsc/src/ts/interface/ts.c
> >>> > [0]PETSC ERROR: #2 TSSolve() line 4374 in
> >>> > /Users/markadams/Codes/petsc/src/ts/interface/ts.c
> >>> > *********** DIVERGED_STEP_REJECTED ierr=91 reason=-2
> >>> > L_2 Diff: 1.32241e+07 at time 0.1025
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >>
> >>> >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d
> >>> ocumentation/faq.html
> >>> >> for
> >>> >> > trouble shooting.
> >>> >> > [0]PETSC ERROR: Petsc Development GIT revision:
> v3.8-53-ga2d93b2d43
> >>> GIT
> >>> >> > Date: 2017-10-09 12:56:49 -0400
> >>> >> > [0]PETSC ERROR: ./ex48 on a arch-macosx-gnu-O named
> MarksMac-5.local
> >>> by
> >>> >> > markadams Wed Nov  1 13:17:15 2017
> >>> >> > [0]PETSC ERROR: Configure options --with-cc=clang
> --with-cc++=clang++
> >>> >> > COPTFLAGS="-O2 -g -mavx2" CXXOPTFLAGS="-O2 -g -mavx2"
> FOPTFLAGS="-O2
> >>> -g
> >>> >> > -mavx2" --download-mpich=1 --download-parmetis=1
> --download-metis=1
> >>> >> > --download-hypre=1 --download-ml=1 --download-triangle=1
> >>> >> > --download-ctetgen=1 --download-p4est=1 --download-fftw
> >>> >> --download-mumps=1
> >>> >> > --download-scalapack=1 --with-x=0 --download-superlu_dist
> >>> >> > --download-superlu --download-ctetgen --with-debugging=0
> >>> >> --download-hdf5=1
> >>> >> > PETSC_ARCH=arch-macosx-gnu-O --download-chaco
> >>> --with-viewfromoptions=1
> >>> >> > [0]PETSC ERROR: #1 TSStep() line 4129 in
> >>> >> > /Users/markadams/Codes/petsc/src/ts/interface/ts.c
> >>> >> > [0]PETSC ERROR: #2 TSSolve() line 4374 in
> >>> >> > /Users/markadams/Codes/petsc/src/ts/interface/ts.c
> >>> >> > [0]PETSC ERROR: #3 main() line 1161 in /Users/markadams/Codes/mhd/
> >>> >> src/ex48.c
> >>> >>
> >>>
> >>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20171101/b3d4360b/attachment.html>