[petsc-dev] Recover from TS failure

[Jed Brown]

Mark Adams <mfadams at lbl.gov> writes:

> On Wed, Nov 1, 2017 at 1:46 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Mark Adams <mfadams at lbl.gov> writes:
>>
>> > I have added some code in a TS post step method to look at the number of
>> > nonlinear iterations and cut the time step if it took too many SNES
>> > iterations.  That helped but now I want to go one step further and
>> recover
>> > from a failed time step (see appended error message).
>> >
>> > How can/should I recover from a SNES failure inside of TS? I could just
>> put
>> > TSSolve in a loop and if TS failed because of the solver, then reduce the
>> > time step and try to recover the state, and try again.
>> >
>> > Thoughts?
>> >
>> > Thanks,
>> > Mark
>> >
>> > Note, this is a direct solver and the linear solver is diverging?  I've
>> > seen the same thing with the maximum its in SNES and failed linesearch.
>> >
>> > 23 TS dt 0.0025 time 0.1025
>> >     0 SNES Function norm 9.997473571601e+04
>> >     1 SNES Function norm 2.295851692533e-01
>> >   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
>> iterations 1
>> > [0]PETSC ERROR: --------------------- Error Message
>> > --------------------------------------------------------------
>> > [0]PETSC ERROR:
>> > [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE,
>> increase
>> > -ts_max_snes_failures or make negative to attempt recovery
>>
>> Did you increase -ts_max_snes_failures (TSSetMaxSNESFailures)?
>>
>
> Ah, read the error message :) with a negative value I get this.
>
> TS can not, or is not going to try, to reconstruct my state and reduce the
> time step. So I think I need to do this myself.

I don't understand.  Why is the linear solve diverging
(-ksp_converged_reason)?  What TS are you using?

> I guess just making loop and testing in the TSConvergedReason, is the way
> to go. We already do something like this for mesh adpativity.
>
> Thanks,
>
> 23 TS dt 0.0025 time 0.1025
>     0 SNES Function norm 9.997473571601e+04
>     1 SNES Function norm 2.295851692533e-01
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 1
>     0 SNES Function norm 9.997473571601e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
>     0 SNES Function norm 9.997473571601e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
>     0 SNES Function norm 9.997473571601e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
>     0 SNES Function norm 9.997473571601e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
>     0 SNES Function norm 9.997473571601e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
>     0 SNES Function norm 9.997473571601e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
>     0 SNES Function norm 9.997473571601e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
>     0 SNES Function norm 9.997473571601e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
>     0 SNES Function norm 9.997473571601e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
>     0 SNES Function norm 9.997473571601e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR:
> [0]PETSC ERROR: TSStep has failed due to DIVERGED_STEP_REJECTED
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
> trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.8-53-ga2d93b2d43  GIT
> Date: 2017-10-09 12:56:49 -0400
> [0]PETSC ERROR: ./ex48 on a arch-macosx-gnu-O named MarksMac-5.local by
> markadams Wed Nov  1 13:51:24 2017
> [0]PETSC ERROR: Configure options --with-cc=clang --with-cc++=clang++
> COPTFLAGS="-O2 -g -mavx2" CXXOPTFLAGS="-O2 -g -mavx2" FOPTFLAGS="-O2 -g
> -mavx2" --download-mpich=1 --download-parmetis=1 --download-metis=1
> --download-hypre=1 --download-ml=1 --download-triangle=1
> --download-ctetgen=1 --download-p4est=1 --download-fftw --download-mumps=1
> --download-scalapack=1 --with-x=0 --download-superlu_dist
> --download-superlu --download-ctetgen --with-debugging=0 --download-hdf5=1
> PETSC_ARCH=arch-macosx-gnu-O --download-chaco --with-viewfromoptions=1
> [0]PETSC ERROR: #1 TSStep() line 4130 in
> /Users/markadams/Codes/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #2 TSSolve() line 4374 in
> /Users/markadams/Codes/petsc/src/ts/interface/ts.c
> *********** DIVERGED_STEP_REJECTED ierr=91 reason=-2
> L_2 Diff: 1.32241e+07 at time 0.1025
>
>
>
>
>>
>> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for
>> > trouble shooting.
>> > [0]PETSC ERROR: Petsc Development GIT revision: v3.8-53-ga2d93b2d43  GIT
>> > Date: 2017-10-09 12:56:49 -0400
>> > [0]PETSC ERROR: ./ex48 on a arch-macosx-gnu-O named MarksMac-5.local by
>> > markadams Wed Nov  1 13:17:15 2017
>> > [0]PETSC ERROR: Configure options --with-cc=clang --with-cc++=clang++
>> > COPTFLAGS="-O2 -g -mavx2" CXXOPTFLAGS="-O2 -g -mavx2" FOPTFLAGS="-O2 -g
>> > -mavx2" --download-mpich=1 --download-parmetis=1 --download-metis=1
>> > --download-hypre=1 --download-ml=1 --download-triangle=1
>> > --download-ctetgen=1 --download-p4est=1 --download-fftw
>> --download-mumps=1
>> > --download-scalapack=1 --with-x=0 --download-superlu_dist
>> > --download-superlu --download-ctetgen --with-debugging=0
>> --download-hdf5=1
>> > PETSC_ARCH=arch-macosx-gnu-O --download-chaco --with-viewfromoptions=1
>> > [0]PETSC ERROR: #1 TSStep() line 4129 in
>> > /Users/markadams/Codes/petsc/src/ts/interface/ts.c
>> > [0]PETSC ERROR: #2 TSSolve() line 4374 in
>> > /Users/markadams/Codes/petsc/src/ts/interface/ts.c
>> > [0]PETSC ERROR: #3 main() line 1161 in /Users/markadams/Codes/mhd/
>> src/ex48.c
>>