<div dir="ltr">I think I found the problem. I configured PETSc with COPTFLAGS=-O3. I'll remove that option and try again.<div><br></div><div>Thanks!</div><div>Harshad</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 14, 2016 at 10:06 AM, Harshad Sahasrabudhe <span dir="ltr"><<a href="mailto:hsahasra@purdue.edu" target="_blank">hsahasra@purdue.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Barry,<div><br></div><div>Thanks for your inputs. I tried to set a watchpoint on ((_p_KSP*)ksp)->reason, but gdb says no symbol _p_KSP in context. Basically, GDB isn't able to find the PETSc source code. I built PETSc with --with-debugging=1 statically and -fPIC, but it seems the libpetsc.a I get doesn't contain debugging symbols (checked using objdump -g). How do I get PETSc library to have debugging info?</div><div><br></div><div>Thanks,</div><div>Harshad</div><div><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 13, 2016 at 2:47 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
> On Sep 13, 2016, at 1:01 PM, Harshad Sahasrabudhe <<a href="mailto:hsahasra@purdue.edu" target="_blank">hsahasra@purdue.edu</a>> wrote:<br>
><br>
> Hi Barry,<br>
><br>
</span><span>> I compiled with mpich configured using --enable-g=meminit to get rid of MPI errors in Valgrind. Doing this reduced the number of errors to 2. I have attached the Valgrind output.<br>
<br>
</span> This isn't helpful but it seems not to be a memory corruption issue :-(<br>
<span>><br>
> I'm using GAMG+GMRES for in each linear iteration of SNES. The linear solver converges with CONVERGED_RTOL for the first 6 iterations and with CONVERGED_STEP_LENGTH after that. I'm still very confused about why this is happening. Any thoughts/ideas?<br>
<br>
</span> Does this happen on one process? If so I would run in the debugger and track the variable to see everyplace the variable is changed, this would point to exactly what piece of code is changing the variable to this unexpected value.<br>
<br>
For example with lldb one can use watch <a href="http://lldb.llvm.org/tutorial.html" rel="noreferrer" target="_blank">http://lldb.llvm.org/tutorial.<wbr>html</a> to see each time a variable gets changed. Similar thing with gdb.<br>
<br>
The variable to watch is ksp->reason Once you get the hang of this it can take just a few minutes to track down the code that is making this unexpected value, though I understand if you haven't done it before it can be intimidating.<br>
<br>
Barry<br>
<br>
You can do the same thing in parallel (like on two processes) if you need to but it is more cumbersome since you need run multiple debuggers. You can have PETSc start up multiple debuggers with mpiexec -n 2 ./ex -start_in_debugger<br>
<div><div><br>
<br>
<br>
<br>
><br>
> Thanks,<br>
> Harshad<br>
><br>
> On Thu, Sep 8, 2016 at 11:26 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> Install your MPI with --download-mpich as a PETSc ./configure option, this will eliminate all the MPICH valgrind errors. Then send as an attachment the resulting valgrind file.<br>
><br>
> I do not 100 % trust any code that produces such valgrind errors.<br>
><br>
> Barry<br>
><br>
><br>
><br>
> > On Sep 8, 2016, at 10:12 PM, Harshad Sahasrabudhe <<a href="mailto:hsahasra@purdue.edu" target="_blank">hsahasra@purdue.edu</a>> wrote:<br>
> ><br>
> > Hi Barry,<br>
> ><br>
> > Thanks for the reply. My code is in C. I ran with Valgrind and found many "Conditional jump or move depends on uninitialized value(s)", "Invalid read" and "Use of uninitialized value" errors. I think all of them are from the libraries I'm using (LibMesh, Boost, MPI, etc.). I'm not really sure what I'm looking for in the Valgrind output. At the end of the file, I get:<br>
> ><br>
> > ==40223== More than 10000000 total errors detected. I'm not reporting any more.<br>
> > ==40223== Final error counts will be inaccurate. Go fix your program!<br>
> > ==40223== Rerun with --error-limit=no to disable this cutoff. Note<br>
> > ==40223== that errors may occur in your program without prior warning from<br>
> > ==40223== Valgrind, because errors are no longer being displayed.<br>
> ><br>
> > Can you give some suggestions on how I should proceed?<br>
> ><br>
> > Thanks,<br>
> > Harshad<br>
> ><br>
> > On Thu, Sep 8, 2016 at 1:59 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
> ><br>
> > This is very odd. CONVERGED_STEP_LENGTH for KSP is very specialized and should never occur with GMRES.<br>
> ><br>
> > Can you run with valgrind to make sure there is no memory corruption? <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html#valgrind</a><br>
> ><br>
> > Is your code fortran or C?<br>
> ><br>
> > Barry<br>
> ><br>
> > > On Sep 8, 2016, at 10:38 AM, Harshad Sahasrabudhe <<a href="mailto:hsahasra@purdue.edu" target="_blank">hsahasra@purdue.edu</a>> wrote:<br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > I'm using GAMG + GMRES for my Poisson problem. The solver converges with KSP_CONVERGED_STEP_LENGTH at a residual of 9.773346857844e-02, which is much higher than what I need (I need a tolerance of at least 1E-8). I am not able to figure out which tolerance I need to set to avoid convergence due to CONVERGED_STEP_LENGTH.<br>
> > ><br>
> > > Any help is appreciated! Output of -ksp_view and -ksp_monitor:<br>
> > ><br>
> > > 0 KSP Residual norm 3.121347818142e+00<br>
> > > 1 KSP Residual norm 9.773346857844e-02<br>
> > > Linear solve converged due to CONVERGED_STEP_LENGTH iterations 1<br>
> > > KSP Object: 1 MPI processes<br>
> > > type: gmres<br>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br>
> > > GMRES: happy breakdown tolerance 1e-30<br>
> > > maximum iterations=10000, initial guess is zero<br>
> > > tolerances: relative=1e-08, absolute=1e-50, divergence=10000<br>
> > > left preconditioning<br>
> > > using PRECONDITIONED norm type for convergence test<br>
> > > PC Object: 1 MPI processes<br>
> > > type: gamg<br>
> > > MG: type is MULTIPLICATIVE, levels=2 cycles=v<br>
> > > Cycles per PCApply=1<br>
> > > Using Galerkin computed coarse grid matrices<br>
> > > Coarse grid solver -- level ------------------------------<wbr>-<br>
> > > KSP Object: (mg_coarse_) 1 MPI processes<br>
> > > type: preonly<br>
> > > maximum iterations=1, initial guess is zero<br>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000<br>
> > > left preconditioning<br>
> > > using NONE norm type for convergence test<br>
> > > PC Object: (mg_coarse_) 1 MPI processes<br>
> > > type: bjacobi<br>
> > > block Jacobi: number of blocks = 1<br>
> > > Local solve is same for all blocks, in the following KSP and PC objects:<br>
> > > KSP Object: (mg_coarse_sub_) 1 MPI processes<br>
> > > type: preonly<br>
> > > maximum iterations=1, initial guess is zero<br>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000<br>
> > > left preconditioning<br>
> > > using NONE norm type for convergence test<br>
> > > PC Object: (mg_coarse_sub_) 1 MPI processes<br>
> > > type: lu<br>
> > > LU: out-of-place factorization<br>
> > > tolerance for zero pivot 2.22045e-14<br>
> > > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]<br>
> > > matrix ordering: nd<br>
> > > factor fill ratio given 5, needed 1.91048<br>
> > > Factored matrix follows:<br>
> > > Mat Object: 1 MPI processes<br>
> > > type: seqaij<br>
> > > rows=284, cols=284<br>
> > > package used to perform factorization: petsc<br>
> > > total: nonzeros=7726, allocated nonzeros=7726<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > using I-node routines: found 133 nodes, limit used is 5<br>
> > > linear system matrix = precond matrix:<br>
> > > Mat Object: 1 MPI processes<br>
> > > type: seqaij<br>
> > > rows=284, cols=284<br>
> > > total: nonzeros=4044, allocated nonzeros=4044<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node routines<br>
> > > linear system matrix = precond matrix:<br>
> > > Mat Object: 1 MPI processes<br>
> > > type: seqaij<br>
> > > rows=284, cols=284<br>
> > > total: nonzeros=4044, allocated nonzeros=4044<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node routines<br>
> > > Down solver (pre-smoother) on level 1 ------------------------------<wbr>-<br>
> > > KSP Object: (mg_levels_1_) 1 MPI processes<br>
> > > type: chebyshev<br>
> > > Chebyshev: eigenvalue estimates: min = 0.195339, max = 4.10212<br>
> > > maximum iterations=2<br>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000<br>
> > > left preconditioning<br>
> > > using nonzero initial guess<br>
> > > using NONE norm type for convergence test<br>
> > > PC Object: (mg_levels_1_) 1 MPI processes<br>
> > > type: sor<br>
> > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1<br>
> > > linear system matrix = precond matrix:<br>
> > > Mat Object: () 1 MPI processes<br>
> > > type: seqaij<br>
> > > rows=9036, cols=9036<br>
> > > total: nonzeros=192256, allocated nonzeros=192256<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node routines<br>
> > > Up solver (post-smoother) same as down solver (pre-smoother)<br>
> > > linear system matrix = precond matrix:<br>
> > > Mat Object: () 1 MPI processes<br>
> > > type: seqaij<br>
> > > rows=9036, cols=9036<br>
> > > total: nonzeros=192256, allocated nonzeros=192256<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node routines<br>
> > ><br>
> > > Thanks,<br>
> > > Harshad<br>
> ><br>
> ><br>
><br>
><br>
</div></div>> <valgrind.log.33199><br>
<br>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>