<div dir="ltr">I am trying to recover from a failed time step with this code. But PETSc does a hard abort with this code:<div><br><div><div> } else {</div><div> /* do not print error messages since process 0 will print them, sleep before aborting so will not accidently kill process 0*/</div><div> PetscSleep(10.0);</div><div> abort();</div><div> }</div></div><div><br></div><div>Can I get TS to not abort and return? </div><div><br></div><div><div> do {</div><div> TSConvergedReason reason;</div><div> ierr = TSGetTime(ts, &old_time);CHKERRQ(ierr);</div><div> ierr = TSSolve(ts, u);</div><div> if (!ierr) break;</div><div> ierr = TSGetConvergedReason(ts,&reason);CHKERRQ(ierr);</div><div> if (reason==TS_DIVERGED_NONLINEAR_SOLVE) { /* recover */</div><div> PetscReal dt;</div><div> ierr = TSGetTime(ts, &t);CHKERRQ(ierr);</div><div> if (t == old_time && failed) {</div><div><span style="white-space:pre"> </span>PetscPrintf(PETSC_COMM_WORLD,"Failed to recover\n");</div><div><span style="white-space:pre"> </span>break; /* failed */</div><div> } else {</div><div><span style="white-space:pre"> </span>old_time = t;</div><div><span style="white-space:pre"> </span>failed = PETSC_TRUE;</div><div> }</div><div> ierr = TSGetTimeStep(ts, &dt);CHKERRQ(ierr);</div><div> ierr = PetscPrintf(PETSC_COMM_WORLD,"recover from %s, reason=%d time=%g, dt=%g --> ",TSConvergedReasons[reason],reason,t,dt);CHKERRQ(ierr);</div><div> dt = dt/ctx.dt_reduction_factor;</div><div> ierr = PetscPrintf(PETSC_COMM_WORLD,"%g\n",dt);CHKERRQ(ierr);</div><div> ierr = TSSetTimeStep(ts, dt);CHKERRQ(ierr);</div><div> ierr = VecCopy(ctx.u0,u);CHKERRQ(ierr); /* recover state */</div><div> } else {</div><div> SETERRQ1(PETSC_COMM_WORLD,PETSC_ERR_ARG_WRONG,"Unhandled error %s",TSConvergedReasons[reason]);</div><div> }</div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 1, 2017 at 2:45 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Wed, Nov 1, 2017 at 2:34 PM, Jed Brown <span dir="ltr"><<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>></span> wrote:<br></span><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> writes:<br>
<br>
> Oh, Maybe the Jacobian has Nans or Infs even though the last time step<br>
> survived. Maybe it was going crazy. I'll check<br>
<br>
</span>If that is the case you would use TSSetFunctionDomainError().<br></blockquote><div><br></div></span><div>Ah, very cool. Good to know. That does not seem to be the problem. Not sure what would cause this linear solver error. I'll try different solvers and see if I get any hints,</div><div>Thanks again,</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="m_7535072333370431434HOEnZb"><div class="m_7535072333370431434h5"><br>
> On Wed, Nov 1, 2017 at 2:13 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
><br>
>> Yea, I don't understand the linear solve error:<br>
>><br>
>> -ts_monitor -ts_type beuler -pc_type lu -pc_factor_mat_solver_package<br>
>> mumps -ksp_type preonly -snes_monitor -snes_rtol 1.e-10 -snes_stol 1.e-10<br>
>> -snes_converged_reason -snes_atol 1.e-18 -snes_converged_reason<br>
>><br>
>> Maybe mumps?<br>
>><br>
>> On Wed, Nov 1, 2017 at 2:02 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>> wrote:<br>
>><br>
>>> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> writes:<br>
>>><br>
>>> > On Wed, Nov 1, 2017 at 1:46 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>> wrote:<br>
>>> ><br>
>>> >> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> writes:<br>
>>> >><br>
>>> >> > I have added some code in a TS post step method to look at the<br>
>>> number of<br>
>>> >> > nonlinear iterations and cut the time step if it took too many SNES<br>
>>> >> > iterations. That helped but now I want to go one step further and<br>
>>> >> recover<br>
>>> >> > from a failed time step (see appended error message).<br>
>>> >> ><br>
>>> >> > How can/should I recover from a SNES failure inside of TS? I could<br>
>>> just<br>
>>> >> put<br>
>>> >> > TSSolve in a loop and if TS failed because of the solver, then<br>
>>> reduce the<br>
>>> >> > time step and try to recover the state, and try again.<br>
>>> >> ><br>
>>> >> > Thoughts?<br>
>>> >> ><br>
>>> >> > Thanks,<br>
>>> >> > Mark<br>
>>> >> ><br>
>>> >> > Note, this is a direct solver and the linear solver is diverging?<br>
>>> I've<br>
>>> >> > seen the same thing with the maximum its in SNES and failed<br>
>>> linesearch.<br>
>>> >> ><br>
>>> >> > 23 TS dt 0.0025 time 0.1025<br>
>>> >> > 0 SNES Function norm 9.997473571601e+04<br>
>>> >> > 1 SNES Function norm 2.295851692533e-01<br>
>>> >> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> >> iterations 1<br>
>>> >> > [0]PETSC ERROR: --------------------- Error Message<br>
>>> >> > ------------------------------<wbr>------------------------------<wbr>--<br>
>>> >> > [0]PETSC ERROR:<br>
>>> >> > [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE,<br>
>>> >> increase<br>
>>> >> > -ts_max_snes_failures or make negative to attempt recovery<br>
>>> >><br>
>>> >> Did you increase -ts_max_snes_failures (TSSetMaxSNESFailures)?<br>
>>> >><br>
>>> ><br>
>>> > Ah, read the error message :) with a negative value I get this.<br>
>>> ><br>
>>> > TS can not, or is not going to try, to reconstruct my state and reduce<br>
>>> the<br>
>>> > time step. So I think I need to do this myself.<br>
>>><br>
>>> I don't understand. Why is the linear solve diverging<br>
>>> (-ksp_converged_reason)? What TS are you using?<br>
>>><br>
>>> > I guess just making loop and testing in the TSConvergedReason, is the<br>
>>> way<br>
>>> > to go. We already do something like this for mesh adpativity.<br>
>>> ><br>
>>> > Thanks,<br>
>>> ><br>
>>> > 23 TS dt 0.0025 time 0.1025<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > 1 SNES Function norm 2.295851692533e-01<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 1<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 0<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 0<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 0<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 0<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 0<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 0<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 0<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 0<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 0<br>
>>> > 0 SNES Function norm 9.997473571601e+04<br>
>>> > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE<br>
>>> iterations 0<br>
>>> > [0]PETSC ERROR: --------------------- Error Message<br>
>>> > ------------------------------<wbr>------------------------------<wbr>--<br>
>>> > [0]PETSC ERROR:<br>
>>> > [0]PETSC ERROR: TSStep has failed due to DIVERGED_STEP_REJECTED<br>
>>> > [0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a><br>
>>> for<br>
>>> > trouble shooting.<br>
>>> > [0]PETSC ERROR: Petsc Development GIT revision: v3.8-53-ga2d93b2d43 GIT<br>
>>> > Date: 2017-10-09 12:56:49 -0400<br>
>>> > [0]PETSC ERROR: ./ex48 on a arch-macosx-gnu-O named MarksMac-5.local by<br>
>>> > markadams Wed Nov 1 13:51:24 2017<br>
>>> > [0]PETSC ERROR: Configure options --with-cc=clang --with-cc++=clang++<br>
>>> > COPTFLAGS="-O2 -g -mavx2" CXXOPTFLAGS="-O2 -g -mavx2" FOPTFLAGS="-O2 -g<br>
>>> > -mavx2" --download-mpich=1 --download-parmetis=1 --download-metis=1<br>
>>> > --download-hypre=1 --download-ml=1 --download-triangle=1<br>
>>> > --download-ctetgen=1 --download-p4est=1 --download-fftw<br>
>>> --download-mumps=1<br>
>>> > --download-scalapack=1 --with-x=0 --download-superlu_dist<br>
>>> > --download-superlu --download-ctetgen --with-debugging=0<br>
>>> --download-hdf5=1<br>
>>> > PETSC_ARCH=arch-macosx-gnu-O --download-chaco --with-viewfromoptions=1<br>
>>> > [0]PETSC ERROR: #1 TSStep() line 4130 in<br>
>>> > /Users/markadams/Codes/petsc/s<wbr>rc/ts/interface/ts.c<br>
>>> > [0]PETSC ERROR: #2 TSSolve() line 4374 in<br>
>>> > /Users/markadams/Codes/petsc/s<wbr>rc/ts/interface/ts.c<br>
>>> > *********** DIVERGED_STEP_REJECTED ierr=91 reason=-2<br>
>>> > L_2 Diff: 1.32241e+07 at time 0.1025<br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> >><br>
>>> >> > [0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/d" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d</a><br>
>>> ocumentation/faq.html<br>
>>> >> for<br>
>>> >> > trouble shooting.<br>
>>> >> > [0]PETSC ERROR: Petsc Development GIT revision: v3.8-53-ga2d93b2d43<br>
>>> GIT<br>
>>> >> > Date: 2017-10-09 12:56:49 -0400<br>
>>> >> > [0]PETSC ERROR: ./ex48 on a arch-macosx-gnu-O named MarksMac-5.local<br>
>>> by<br>
>>> >> > markadams Wed Nov 1 13:17:15 2017<br>
>>> >> > [0]PETSC ERROR: Configure options --with-cc=clang --with-cc++=clang++<br>
>>> >> > COPTFLAGS="-O2 -g -mavx2" CXXOPTFLAGS="-O2 -g -mavx2" FOPTFLAGS="-O2<br>
>>> -g<br>
>>> >> > -mavx2" --download-mpich=1 --download-parmetis=1 --download-metis=1<br>
>>> >> > --download-hypre=1 --download-ml=1 --download-triangle=1<br>
>>> >> > --download-ctetgen=1 --download-p4est=1 --download-fftw<br>
>>> >> --download-mumps=1<br>
>>> >> > --download-scalapack=1 --with-x=0 --download-superlu_dist<br>
>>> >> > --download-superlu --download-ctetgen --with-debugging=0<br>
>>> >> --download-hdf5=1<br>
>>> >> > PETSC_ARCH=arch-macosx-gnu-O --download-chaco<br>
>>> --with-viewfromoptions=1<br>
>>> >> > [0]PETSC ERROR: #1 TSStep() line 4129 in<br>
>>> >> > /Users/markadams/Codes/petsc/s<wbr>rc/ts/interface/ts.c<br>
>>> >> > [0]PETSC ERROR: #2 TSSolve() line 4374 in<br>
>>> >> > /Users/markadams/Codes/petsc/s<wbr>rc/ts/interface/ts.c<br>
>>> >> > [0]PETSC ERROR: #3 main() line 1161 in /Users/markadams/Codes/mhd/<br>
>>> >> src/ex48.c<br>
>>> >><br>
>>><br>
>><br>
>><br>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>