<div dir="ltr">This is comming from blas. How did you configure blas?</div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 10, 2017 at 10:38 AM, Hong <span dir="ltr"><<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Using petsc machine, I get<div><div>hzhang@petsc /sandbox/hzhang/petsc/src/<wbr>snes/examples/tutorials (hzhang/fix-submat_<wbr>samerowdist)</div><div>$ mpiexec -n 2 valgrind ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_<wbr>esteig 0,0.05,0,1.05 -mg_levels_pc_type jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1</div></div><div><br></div><div><div>==28811== Memcheck, a memory error detector</div><div>==28811== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.</div><div>==28811== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info</div><div>==28811== Command: ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_<wbr>esteig 0,0.05,0,1.05 -mg_levels_pc_type jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1</div></div><div>...</div><div><div>==28811== Invalid read of size 16</div><div>==28811== at 0x8550946: dswap_k_NEHALEM (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==28811== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==28811== by 0x75B33B2: dgetri_ (in /usr/lib/lapack/liblapack.so.<wbr>3.0)</div><div>==28811== by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012)</div><div>==28811== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)</div><div>==28811== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)</div><div>==28811== by 0x41056E: main (ex56.c:395)</div><div>==28811== Address 0xdc650d0 is 52,480 bytes inside a block of size 52,488 alloc'd</div><div>==28811== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_<wbr>memcheck-amd64-linux.so)</div><div>==28811== by 0x51590F6: PetscMallocAlign (mal.c:39)</div><div>==28811== by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983)</div><div>==28811== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)</div><div>==28811== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)</div><div>==28811== by 0x41056E: main (ex56.c:395)</div><div>==28811==</div><div>==28812== Invalid read of size 16</div><div>==28812== at 0x8550946: dswap_k_NEHALEM (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==28812== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==28812== by 0x75B33B2: dgetri_ (in /usr/lib/lapack/liblapack.so.<wbr>3.0)</div><div>==28812== by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012)</div><div>==28812== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)</div><div>==28812== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)</div><div>==28812== by 0x41056E: main (ex56.c:395)</div><div>==28812== Address 0xd9c7600 is 52,480 bytes inside a block of size 52,488 alloc'd</div><div>==28812== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_<wbr>memcheck-amd64-linux.so)</div><div>==28812== by 0x51590F6: PetscMallocAlign (mal.c:39)</div><div>==28812== by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983)</div><div>==28812== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)</div><div>==28812== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)</div><div>==28812== by 0x41056E: main (ex56.c:395)</div><div>==28812==</div><div>==28811== Invalid read of size 16</div><div>==28811== at 0x8550A55: dswap_k_NEHALEM (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==28811== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==28811== by 0x7675179: dsteqr_ (in /usr/lib/lapack/liblapack.so.<wbr>3.0)</div><div>==28811== by 0x5DFFA22: PetscDTGaussQuadrature (dt.c:508)</div><div>==28811== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)</div><div>==28811== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)</div><div>==28811== by 0x41056E: main (ex56.c:395)</div><div>==28811== Address 0xd99cbe0 is 64 bytes inside a block of size 72 alloc'd</div><div>==28811== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_<wbr>memcheck-amd64-linux.so)</div><div>==28811== by 0x51590F6: PetscMallocAlign (mal.c:39)</div><div>==28811== by 0x5DFF766: PetscDTGaussQuadrature (dt.c:504)</div><div>==28811== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)</div><div>==28811== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)</div><div>==28811== by 0x41056E: main (ex56.c:395)</div><div>==28811==</div><div>==28812== Invalid read of size 16</div><div>==28812== at 0x8550A55: dswap_k_NEHALEM (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==28812== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/<wbr>libblas.so.3)</div><div>==28812== by 0x7675179: dsteqr_ (in /usr/lib/lapack/liblapack.so.<wbr>3.0)</div><div>==28812== by 0x5DFFA22: PetscDTGaussQuadrature (dt.c:508)</div><div>==28812== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)</div><div>==28812== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)</div><div>==28812== by 0x41056E: main (ex56.c:395)</div><div>==28812== Address 0xdc11f30 is 64 bytes inside a block of size 72 alloc'd</div><div>==28812== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_<wbr>memcheck-amd64-linux.so)</div><div>==28812== by 0x51590F6: PetscMallocAlign (mal.c:39)</div><div>==28812== by 0x5DFF766: PetscDTGaussQuadrature (dt.c:504)</div><div>==28812== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)</div><div>==28812== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)</div><div>==28812== by 0x41056E: main (ex56.c:395)</div><div>==28812==</div><span class=""><div>[0] 27 global equations, 9 vertices</div><div>[0] 27 equations in vector, 9 vertices</div><div>Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1</div><div>[0] 441 global equations, 147 vertices</div><div>[0] 441 equations in vector, 147 vertices</div></span></div><div><br></div><div>hangs here ...</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Hong</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 10, 2017 at 8:57 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This printed a little funny in gmail, snes/ex56 is running clean in the first few loops (appended), but the last one is the one with a reduced processor set. Still waiting. This is with 32 bit integers. I'm running another with 64 bit integers.<div><br></div><div> ...</div><span><div><div>[0] 27 global equations, 9 vertices</div><div>[0] 27 equations in vector, 9 vertices</div><div>Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1</div><div>[0] 441 global equations, 147 vertices</div><div>[0] 441 equations in vector, 147 vertices</div><div>Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1</div><div>[0] 4725 global equations, 1575 vertices</div><div>[0] 4725 equations in vector, 1575 vertices</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></span></div><div class="m_6780774247923759259HOEnZb"><div class="m_6780774247923759259h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 10, 2017 at 9:06 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Thu, Nov 9, 2017 at 1:56 PM, Hong <span dir="ltr"><<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Mark:<span><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">OK, well, just go with the Linux machine for the regression test. I will keep trying to reproduce this on my Mac with an O build.</div></blockquote><div> </div></span><span><div>Valgrind error occurs on linux machines with g-build. I cannot merge this branch to maint until the bug is fixed. </div></span></div></div></div></blockquote><div><br></div><div>Valgrind is failing on this run on my Mac. Moving to cg, like you I suppose. This takes forever. This is what I have so far. Did you get this far?</div><div><br></div><div><div>07:48 hzhang/fix-submat_samerowdist *= /sandbox/adams/petsc/src/snes/<wbr>examples/tutorials$ make PETSC_DIR=/sandbox/adams/petsc PETSC_ARCH=arch-linux2-c-dbg32 val</div></div><div>/sandbox/adams/petsc/arch-linu<wbr>x2-c-dbg32/bin/mpiexec -n 2 valgrind ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_estei<wbr>g 0,0.05,0,1.05 -mg_levels_pc_type jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1<br></div><div><div>==12414== Memcheck, a memory error detector</div><div>==12414== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.</div><div>==12414== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info</div><div>==12415== Memcheck, a memory error detector</div><div>==12415== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.</div><div>==12415== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info</div><div>==12415== Command: ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_estei<wbr>g 0,0.05,0,1.05 -mg_levels_pc_type jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1</div><div>==12415== </div><div>==12414== Command: ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_estei<wbr>g 0,0.05,0,1.05 -mg_levels_pc_type jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1</div><div>==12414== </div><span><div>[0] 27 global equations, 9 vertices</div><div>[0] 27 equations in vector, 9 vertices</div></span><span><div>Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1</div></span><span><div>[0] 441 global equations, 147 vertices</div><div>[0] 441 equations in vector, 147 vertices</div></span><span><div>Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1</div></span><span><div>[0] 4725 global equations, 1575 vertices</div><div>[0] 4725 equations in vector, 1575 vertices</div></span></div><div><br></div><div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>