<div><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Mon, 6 May 2019 at 02:18, Smith, Barry F. via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
Even if you don't get failures on the smaller version of a code it can still be worth running with valgrind (when you can't run valgrind on the massive problem) because often the problem is still there on the smaller problem, just less directly visible but valgrind can still find it.<br>
<br>
<br>
> [13]PETSC ERROR: Object is in wrong state<br>
> [13]PETSC ERROR: Clearing DM of global vectors that has a global vector obtained with DMGetGlobalVector()<br>
<br>
You probably have a work vector obtained with DMGetGlobalVector() that you forgot to return with DMRestoreGlobalVector(). Though I would expect that this would reproduce on any size problem.</blockquote><div dir="auto"><br></div><div dir="auto">I'd fix the DM issue first before addressing the solver problem. I suspect the DM error could cause the solver error.</div><div dir="auto"><br></div><div dir="auto">Yep - something is wrong with your management of vectors associated with one of your DM's. You can figure out if this is the case by running with -log_view. Make sure the summary of the objects reported shows that the number of Vecs created and destroyed matches. At the very least, if there is a mismatch, make sure this difference does not increase as you do additional optimization solvers (or time steps).</div><div dir="auto"><br></div><div dir="auto">As Barry says, you don't need to run a large scale job to detect this, nor do you need to run through many optimization solves - the problem exists and is detectable and thus fixable for all job sizes.</div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
Barry<br>
<br>
<br>
> On May 5, 2019, at 5:21 PM, Randall Mackie via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> <br>
> In solving a nonlinear optimization problem, I was recently experimenting with fgmres using the following options:<br>
> <br>
> -nlcg_ksp_type fgmres \<br>
> -nlcg_pc_type ksp \<br>
> -nlcg_ksp_ksp_type bcgs \<br>
> -nlcg_ksp_pc_type jacobi \<br>
> -nlcg_ksp_rtol 1e-6 \<br>
> -nlcg_ksp_ksp_max_it 300 \<br>
> -nlcg_ksp_max_it 200 \<br>
> -nlcg_ksp_converged_reason \<br>
> -nlcg_ksp_monitor_true_residual \<br>
> <br>
> I sometimes randomly will get an error like the following:<br>
> <br>
> Residual norms for nlcg_ solve.<br>
> 0 KSP unpreconditioned resid norm 3.371606868500e+04 true resid norm 3.371606868500e+04 ||r(i)||/||b|| 1.000000000000e+00<br>
> 1 KSP unpreconditioned resid norm 2.322590778002e+02 true resid norm 2.322590778002e+02 ||r(i)||/||b|| 6.888676137487e-03<br>
> 2 KSP unpreconditioned resid norm 8.262440884758e+01 true resid norm 8.262440884758e+01 ||r(i)||/||b|| 2.450594392232e-03<br>
> 3 KSP unpreconditioned resid norm 3.660428333809e+01 true resid norm 3.660428333809e+01 ||r(i)||/||b|| 1.085662853522e-03<br>
> 3 KSP unpreconditioned resid norm 0.000000000000e+00 true resid norm -nan ||r(i)||/||b|| -nan<br>
> Linear nlcg_ solve did not converge due to DIVERGED_PC_FAILED iterations 3<br>
> PC_FAILED due to SUBPC_ERROR <br>
> <br>
> This usually happens after a few nonlinear optimization iterations, meaning that it’s worked perfectly fine until this point.<br>
> How can using jacobi pc all of a sudden cause a NaN, if it’s worked perfectly fine before?<br>
> <br>
> Some other errors in the output log file are as follows, although I have no idea if they result from the above error or not:<br>
> <br>
> [13]PETSC ERROR: Object is in wrong state<br>
> [13]PETSC ERROR: Clearing DM of global vectors that has a global vector obtained with DMGetGlobalVector()<br>
> [13]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
> [13]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019<br>
> <br>
> <br>
> [27]PETSC ERROR: #1 DMClearGlobalVectors() line 196 in /state/std2/FEMI/PETSc/petsc-3.11.1/src/dm/interface/dmget.c<br>
> [27]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex --with-debugging=0 --with-fortran=1 --with-blaslapack-dir=/state/std2/intel_2018/m<br>
> kl --with-mkl_pardiso-dir=/state/std2/intel_2018/mkl --with-mkl_cpardiso-dir=/state/std2/intel_2018/mkl --download-mumps=../external/mumps_v5.1.2-p1.tar.gz --d<br>
> ownload-scalapack=../external/scalapack-2.0.2.tgz --with-cc=mpiicc --with-fc=mpiifort --with-cxx=mpiicc --FOPTFLAGS="-O3 -xHost" --COPTFLAGS="-O3 -xHost" --CXX<br>
> OPTFLAGS="-O3 -xHost"<br>
> <br>
> <br>
> #2 DMDestroy() line 752 in /state/std2/FEMI/PETSc/petsc-3.11.1/src/dm/interface/dm.c<br>
> [72]PETSC ERROR: #3 PetscObjectDereference() line 624 in /state/std2/FEMI/PETSc/petsc-3.11.1/src/sys/objects/inherit.c<br>
> [72]PETSC ERROR: #4 PetscObjectListDestroy() line 156 in /state/std2/FEMI/PETSc/petsc-3.11.1/src/sys/objects/olist.c<br>
> [72]PETSC ERROR: #5 PetscHeaderDestroy_Private() line 122 in /state/std2/FEMI/PETSc/petsc-3.11.1/src/sys/objects/inherit.c<br>
> [72]PETSC ERROR: #6 VecDestroy() line 412 in /state/std2/FEMI/PETSc/petsc-3.11.1/src/vec/vec/interface/vector.c<br>
> <br>
> <br>
> <br>
> This is a large run taking many hours to get to this problem. I will try to run in debug mode, but given that this seems to be randomly happening (this has happened maybe 30% of the time I have used the fgmres option), there is no guarantee that will show anything useful. Valgrind is obviously out of the question for a large run, and I have yet to reproduce this on a smaller run.<br>
> <br>
> Anyone have any ideas as to what’s causing this?<br>
> <br>
> Thanks in advance,<br>
> <br>
> Randy M.<br>
<br>
</blockquote></div></div>