<div dir="ltr"> I found the bug. This Linux compiler did give me a warning about a possible uninitialized variable that is used for normalization, with a slightly convoluted variable initialization. Apparently some compilers reorganized this loop and stuff was normalized with an uninitialized variable.<div><br></div><div>Anyway, thanks again for getting me setup on cg.</div><div>Mark</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 15, 2016 at 9:06 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Damn, it runs pretty clean, and the code runs, even with optimized. This one warning goes away with -malloc_debug -malloc_dump (appended). This is the source of the valgrind warning. This is new code with multi-species that I just added. Am I making a mistake here?<div><br></div><div> ierr = DMGetDS(ctx->dm, &prob);CHKERRQ(ierr);<br></div><div><div> for (ii=0;ii<ctx->num_species;ii++<wbr>) {</div><div> char buf[256];</div><div> if (ii==0) ierr = PetscSNPrintf(buf, 256, "e");</div><div> else ierr = PetscSNPrintf(buf, 256, "i%D", ii);</div><div> CHKERRQ(ierr);</div><div> /* Setup Discretization - FEM */</div><div> ierr = PetscFECreateDefault(ctx->dm, dim, 1, ctx->simplex, NULL, PETSC_DECIDE, &ctx->fe[ii]);CHKERRQ(ierr);</div><div> ierr = LandFEChangeQuadrature(ctx-><wbr>fe[ii]);CHKERRQ(ierr);</div><div> ierr = PetscObjectSetName((<wbr>PetscObject) ctx->fe[ii], buf);CHKERRQ(ierr);</div><div> ierr = PetscDSSetDiscretization(prob, ii, (PetscObject) ctx->fe[ii]);CHKERRQ(ierr); /* this is where Nf gets set */</div><div> }</div></div><div><br></div><div>Thanks,</div><div><br></div><div><div>/sandbox/adams/petsc/arch-<wbr>linux2-c-debug/bin/mpiexec -n 1 valgrind --dsymutil=yes --leak-check=no --gen-suppressions=no --num-callers=20 --error-limit=no ./landaufem -dm_refine 2 -snes_rtol 5.e-8 -snes_stol 5.e-8 -ts_type theta -ts_theta_theta 0.5 -ts_theta_endpoint -theta .05 -pc_type lu -petscspace_order 2 -mass_petscspace_order 2 -petscspace_poly_tensor 1 -mass_petscspace_poly_tensor 1 -dt 1.e-3 -tsteps 1 -verbose 2 -num_species 2 -snes_monitor -plot_file_prefix t</div><div>==26477== Memcheck, a memory error detector</div><div>==26477== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.</div><div>==26477== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info</div><div>==26477== Command: ./landaufem -dm_refine 2 -snes_rtol 5.e-8 -snes_stol 5.e-8 -ts_type theta -ts_theta_theta 0.5 -ts_theta_endpoint -theta .05 -pc_type lu -petscspace_order 2 -mass_petscspace_order 2 -petscspace_poly_tensor 1 -mass_petscspace_poly_tensor 1 -dt 1.e-3 -tsteps 1 -verbose 2 -num_species 2 -snes_monitor -plot_file_prefix t</div><div>==26477== </div><div>==26477== Thread 9:</div><div>==26477== Invalid read of size 16</div><div>==26477== at 0x9376634: dswap_k_SANDYBRIDGE (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==26477== by 0x835AE60: ??? (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==26477== by 0x835B14C: ??? (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==26477== by 0xA100183: start_thread (pthread_create.c:312)</div><div>==26477== by 0x751837C: clone (clone.S:111)</div><div>==26477== Address 0x93bece50 is 640 bytes inside a block of size 648 alloc'd</div><div>==26477== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_<wbr>memcheck-amd64-linux.so)</div><div>==26477== by 0x4ED5959: PetscMallocAlign (mal.c:28)</div><div>==26477== by 0x5CD12B0: PetscFESetUp_Basic (dtfe.c:3847)</div><div>==26477== by 0x5CC69ED: PetscFESetUp (dtfe.c:3094)</div><div>==26477== by 0x5CEBB70: PetscFECreateDefault (dtfe.c:6606)</div><div>==26477== by 0x40DEB1: main (main.c:775)</div><div>==26477== </div><div> *** LandFEChangeQuadrature No hack tensor=1 Nq=9 dim=2 ***</div><div> *** LandFEChangeQuadrature No hack tensor=1 Nq=9 dim=2 ***</div><div> *** LandFEChangeQuadrature No hack tensor=1 Nq=9 dim=2 ***</div><div> *** LandFEChangeQuadrature No hack tensor=1 Nq=9 dim=2 ***</div><div> 0) species 0: momentum= -3.53244532544986e-02, energy= 1.51054814051742e-02</div><div> 0) species 1: momentum= 7.06489065089972e-02, energy= 3.02109628103485e-02</div><div> 0) Total momentum= 3.53244532544986e-02, energy= 4.53164442155227e-02 (m_i/m_e = 2.)</div><div> 0 SNES Function norm 4.438513482798e+01 </div><div> 1 SNES Function norm 1.644115258674e+00 </div><div> 2 SNES Function norm 1.246811010563e-01 </div><div> 3 SNES Function norm 1.109117454301e-02 </div><div> 4 SNES Function norm 1.006853809120e-03 </div><div> 5 SNES Function norm 9.334138459020e-05 </div><div> 6 SNES Function norm 8.543152498171e-06 </div><div> 7 SNES Function norm 7.831310487414e-07 </div><div> 1) species 0: momentum= -3.18205425912776e-02, energy= 1.57071588338379e-02</div><div> 1) species 1: momentum= 6.71449958176972e-02, energy= 2.96092852424931e-02</div><div> 1) Total momentum= 3.53244532264196e-02, energy= 4.53164440763310e-02 (m_i/m_e = 2.)</div><div>1 TS time steps, 32 cells, Nq=9 (288 IPs), T=0.001</div><div>==26477== </div><div>==26477== HEAP SUMMARY:</div><div>==26477== in use at exit: 0 bytes in 0 blocks</div><div>==26477== total heap usage: 8,618 allocs, 8,556 frees, 6,754,155 bytes allocated</div><div>==26477== </div><div>==26477== For a detailed leak analysis, rerun with: --leak-check=full</div><div>==26477== </div><div>==26477== For counts of detected and suppressed errors, rerun with: -v</div><div>==26477== ERROR SUMMARY: 12 errors from 1 contexts (suppressed: 0 from 0)</div></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 15, 2016 at 8:10 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Great, thanks, all installed and configuring.<div><br></div><div>Humm, and I forgot to do PETSC_DIR=$PWD ...</div></div><div class="m_3935220467960818506HOEnZb"><div class="m_3935220467960818506h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 15, 2016 at 6:43 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Mark,<br>
<br>
My records indicate you have accounts at MCS, at least we are paying for them :-). You should be able to access them<br>
with<br>
<br>
ssh <a href="mailto:adams@login.mcs.anl.gov" target="_blank">adams@login.mcs.anl.gov</a><br>
<br>
then<br>
<br>
ssh cg<br>
<br>
then<br>
<br>
cd /sandbox/<br>
mkdir adams<br>
cd adams<br>
git clone git@bitbucket.org:petsc/petsc.<wbr>git<br>
cd petsc<br>
./configure --download-mpich<br>
<br>
If you never set your ssh key to login then you need to do it at <a href="http://accounts.mcs.anl.gov" rel="noreferrer" target="_blank">accounts.mcs.anl.gov</a> (note that you cannot ssh via a password you need to set the ssh key).<br>
<span class="m_3935220467960818506m_4443818315811748081HOEnZb"><font color="#888888"><br>
Barry<br>
</font></span><div class="m_3935220467960818506m_4443818315811748081HOEnZb"><div class="m_3935220467960818506m_4443818315811748081h5"><br>
> On Dec 15, 2016, at 5:32 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
><br>
> OK, that was useful. I don't have access to a Linux machine right now. I must have an uninitialized variable (that the compiler did not catch).<br>
> Thanks,<br>
><br>
> [0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [0]PETSC ERROR:<br>
> [0]PETSC ERROR: SNESSolve has not converged due to Nan or Inf norm<br>
> [0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.<br>
> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.4-2475-g4fbd04f GIT Date: 2016-12-11 10:34:33 -0500<br>
> [0]PETSC ERROR: /global/u2/m/madams/landaufem/<wbr>Plex/./landaufem on a arch-xc30-opt64-intel named nid00012 by madams Thu Dec 15 15:31:15 2016<br>
> [0]PETSC ERROR: Configure options COPTFLAGS="-fast -no-ipo -g" CXXOPTFLAGS="-fast -no-ipo -g" FOPTFLAGS="-fast -no-ipo -g" --download-hypre --download-parmetis --download-metis --download-p4est --with-hdf5-dir=/opt/cray/hdf5<wbr>-parallel/1.8.16/INTEL/15.0 --with-ssl=0 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0 --with-x=0 --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices PETSC_ARCH=arch-xc30-opt64-int<wbr>el<br>
> [0]PETSC ERROR: #1 SNESSolve_NEWTONLS() line 186 in /global/u2/m/madams/petsc/src/<wbr>snes/impls/ls/ls.c<br>
> TSSolve failed<br>
> 0 TS time steps, 528 cells, Nq=7 (3696 IPs), T=0.<br>
> [0]PETSC ERROR: #2 SNESSolve() line 4128 in /global/u2/m/madams/petsc/src/<wbr>snes/interface/snes.c<br>
> [0]PETSC ERROR: #3 TS_SNESSolve() line 189 in /global/u2/m/madams/petsc/src/<wbr>ts/impls/implicit/theta/theta.<wbr>c<br>
><br>
><br>
> On Thu, Dec 15, 2016 at 6:24 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> It is useful to run with valgrind, even on a completely different machine, when you have errors because it will detect any memory corruption. So currently I do my valgrind runs on a linux machine.<br>
><br>
> Assuming the prefix is correct -snes_monitor should print an initial residual norm before doing any solves so it is curious you got no output. You can run with -snes_error_if_not_converged -ksp_error_if_not_converged to try to get it to output as soon as a problem is detected.<br>
><br>
> Barry<br>
><br>
><br>
> > On Dec 15, 2016, at 4:56 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> ><br>
> > I have a code that work on my Mac but it fails on both a Cray XC30 and a KNL, unless the code is build with debug. I get this error message + -info output. I am using -snes_monitor but get no output. This code was working and I added a new feature. It does seem to fail when this new feature is used.<br>
> ><br>
> > And, alas, I do not have a functioning valgrind right now.<br>
> ><br>
> > I will start toggling optimization flags but any thoughts would be welcome.<br>
> ><br>
> > Mark<br>
> ><br>
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4290 X 4290; storage space: 0 unneeded,132612 used<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 50<br>
> > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 4290) < 0.6. Do not use CompressedRow routines.<br>
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374784<br>
> > [0] DMGetDMKSP(): Creating new DMKSP<br>
> > [0] TSAdaptCheckStage(): Step=0, nonlinear solve failures 1 greater than current TS allowed, stopping solve<br>
> > [0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> > [0]PETSC ERROR:<br>
> > [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, increase -ts_max_snes_failures or make negative to attempt recovery<br>
> > [0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.<br>
> > [0]PETSC ERROR: Petsc Development GIT revision: unknown GIT Date: unknown<br>
> > [0]PETSC ERROR: /global/u2/m/madams/landaufem/<wbr>Plex/./landaufem on a arch-cori-knl-opt64-novector-i<wbr>ntel named nid09355 by madams Thu Dec 15 14:25:00 2016<br>
> > [0]PETSC ERROR: TSSolve failed<br>
> > 1 TS time steps, 512 cells, Nq=9 (4608 IPs), T=0.<br>
> > Configure options COPTFLAGS=" -g -O1 -fp-model fast -qopt-report=5 -hcpu=mic-knl -no-simd" CXXOPTFLAGS="-g -O1 -fp-model fast -qopt-report=5 -hcpu=mic-knl -no-simd" FOPTFLAGS=" -g -O1 -fp-model fast -qopt-report=5 -hcpu=mic-knl -no-simd" --download-metis=1 --download-parmetis=1 --with-blas-lapack-dir=/usr/co<wbr>mmon/software/intel/compilers_<wbr>and_libraries_2016.3.210/linux<wbr>/mkl --with-cc=mpiicc --with-cxx=mpiicpc --with-debugging=0 --with-fc=mpiifort --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices PETSC_ARCH=arch-cori-knl-opt64<wbr>-novector-intel --with-openmp=0 --download-p4est=0<br>
> > [0]PETSC ERROR: #1 TSStep() line 3972 in /global/u2/m/madams/petsc/src/<wbr>ts/interface/ts.c<br>
> > [0]PETSC ERROR: #2 TSSolve() line 4218 in /global/u2/m/madams/petsc/src/<wbr>ts/interface/ts.c<br>
> ><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>