[petsc-users] unsorted local columns in 3.8?

Fri Nov 10 19:40:17 CST 2017

On Fri, Nov 10, 2017 at 6:09 PM, Hong <hzhang at mcs.anl.gov> wrote:

> Satish :
>
>> I don't see any changes hzhang/fix-submat_samerowdist in past few
>> days. So how did this error get fixed?
>>
>
> I've never seen valgrind error for ex56 from nightly tests.
> Mark told me about Valgrind error on his mac,
>

Valgrind segv'ed on my Mac. (This makes me suspicious because valgrind has
been working OK for my on my Mac, but let move on.) I ran on Linux (CG at
ANL) and it ran fine.

The error that Hong is seeing is in LAPACK. I download LAPACK (on CG, not
my Mac) but Hong does not seem to be downloading LAPACK. So I think that
the lapack that configure is picking up throwing these valgrind warnings.

and I reproduced it on our linux machine with my own configure,
>
which directs to lapack.
>
>>
>> please do not merge yet.
>>
> Sure.
>
> Hong
>
>>
>> On Fri, 10 Nov 2017, Mark Adams wrote:
>>
>> > I don't see a problem with this. As far as what branch to merge it into,
>> > that is a Barry/Satish/etc decision.
>> >
>> > On Fri, Nov 10, 2017 at 5:17 PM, Hong <hzhang at mcs.anl.gov> wrote:
>> >
>> > > Mark:
>> > >
>> > >> I would add:
>> > >>
>> > >>     '--download-fblaslapack=1',
>> > >>
>> > >> This is what I have on my Linux machine (cg at ANL) and it runs
>> clean.
>> > >>
>> > > I do not see any error report from nightly tests for ex56. Should I
>> merge
>> > > this branch to maint?
>> > > Hong
>> > >
>> > >>
>> > >> On Fri, Nov 10, 2017 at 11:03 AM, Hong <hzhang at mcs.anl.gov> wrote:
>> > >>
>> > >>> I use
>> > >>> Using configure Options: --download-metis --download-mpich
>> > >>> --download-mumps --download-parmetis --download-scalapack
>> > >>> --download-superlu --download-superlu_dist --download-suitesparse
>> > >>> --download-hypre --download-ptscotch --download-chaco
>> --with-ctable=1
>> > >>> --download-cmake --with-cc=gcc --with-cxx=g++ --with-debugging=1
>> > >>> --with-visibility=0 --with-fc=gfortran
>> > >>> Hong
>> > >>>
>> > >>> On Fri, Nov 10, 2017 at 9:59 AM, Mark Adams <mfadams at lbl.gov>
>> wrote:
>> > >>>
>> > >>>> This must be a configure issue. I don't see these warning:
>> > >>>>
>> > >>>> #!/usr/bin/python
>> > >>>> if __name__ == '__main__':
>> > >>>>   import sys
>> > >>>>   import os
>> > >>>>   sys.path.insert(0, os.path.abspath('config'))
>> > >>>>   import configure
>> > >>>>   configure_options = [
>> > >>>>     '--with-cc=clang',
>> > >>>>     '--with-cc++=clang++',
>> > >>>>     '--download-mpich=1',
>> > >>>>     '--download-metis=1',
>> > >>>>     '--download-superlu=1',
>> > >>>>     '--download-superlu_dist=1',
>> > >>>>     '--download-parmetis=1',
>> > >>>>     '--download-fblaslapack=1',
>> > >>>>     '--download-p4est=1',
>> > >>>>     '--with-debugging=1',
>> > >>>>     '--with-batch=0',
>> > >>>>     'PETSC_ARCH=arch-linux2-c-dbg32',
>> > >>>>     '--with-openmp=0',
>> > >>>>     '--download-p4est=0'
>> > >>>>   ]
>> > >>>>   configure.petsc_configure(configure_options)
>> > >>>>
>> > >>>> ~
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> On Fri, Nov 10, 2017 at 10:56 AM, Mark Adams <mfadams at lbl.gov>
>> wrote:
>> > >>>>
>> > >>>>> This is comming from blas. How did you configure blas?
>> > >>>>>
>> > >>>>> On Fri, Nov 10, 2017 at 10:38 AM, Hong <hzhang at mcs.anl.gov>
>> wrote:
>> > >>>>>
>> > >>>>>> Using petsc machine, I get
>> > >>>>>> hzhang at petsc /sandbox/hzhang/petsc/src/snes/examples/tutorials
>> > >>>>>> (hzhang/fix-submat_samerowdist)
>> > >>>>>> $ mpiexec -n 2 valgrind ./ex56 -cells 2,2,1 -max_conv_its 3
>> > >>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg
>> -ksp_rtol
>> > >>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10
>> -pc_type gamg
>> > >>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1
>> -pc_gamg_coarse_eq_limit 10
>> > >>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>> > >>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
>> -snes_converged_reason
>> > >>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1
>> -mg_levels_ksp_type
>> > >>>>>> chebyshev -mg_levels_esteig_ksp_type cg
>> -mg_levels_esteig_ksp_max_it 10
>> > >>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>> > >>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3
>> -run_type 1
>> > >>>>>>
>> > >>>>>> ==28811== Memcheck, a memory error detector
>> > >>>>>> ==28811== Copyright (C) 2002-2013, and GNU GPL'd, by Julian
>> Seward et
>> > >>>>>> al.
>> > >>>>>> ==28811== Using Valgrind-3.10.1 and LibVEX; rerun with -h for
>> > >>>>>> copyright info
>> > >>>>>> ==28811== Command: ./ex56 -cells 2,2,1 -max_conv_its 3
>> > >>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg
>> -ksp_rtol
>> > >>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10
>> -pc_type gamg
>> > >>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1
>> -pc_gamg_coarse_eq_limit 10
>> > >>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>> > >>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
>> -snes_converged_reason
>> > >>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1
>> -mg_levels_ksp_type
>> > >>>>>> chebyshev -mg_levels_esteig_ksp_type cg
>> -mg_levels_esteig_ksp_max_it 10
>> > >>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>> > >>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3
>> -run_type 1
>> > >>>>>> ...
>> > >>>>>> ==28811== Invalid read of size 16
>> > >>>>>> ==28811==    at 0x8550946: dswap_k_NEHALEM (in
>> > >>>>>> /usr/lib/openblas-base/libblas.so.3)
>> > >>>>>> ==28811==    by 0x7C6797F: dswap_ (in
>> /usr/lib/openblas-base/libblas
>> > >>>>>> .so.3)
>> > >>>>>> ==28811==    by 0x75B33B2: dgetri_ (in
>> /usr/lib/lapack/liblapack.so.3
>> > >>>>>> .0)
>> > >>>>>> ==28811==    by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012)
>> > >>>>>> ==28811==    by 0x5E320C9: PetscFESetUp (dtfe.c:3274)
>> > >>>>>> ==28811==    by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)
>> > >>>>>> ==28811==    by 0x41056E: main (ex56.c:395)
>> > >>>>>> ==28811==  Address 0xdc650d0 is 52,480 bytes inside a block of
>> size
>> > >>>>>> 52,488 alloc'd
>> > >>>>>> ==28811==    at 0x4C2D110: memalign (in
>> /usr/lib/valgrind/vgpreload_me
>> > >>>>>> mcheck-amd64-linux.so)
>> > >>>>>> ==28811==    by 0x51590F6: PetscMallocAlign (mal.c:39)
>> > >>>>>> ==28811==    by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983)
>> > >>>>>> ==28811==    by 0x5E320C9: PetscFESetUp (dtfe.c:3274)
>> > >>>>>> ==28811==    by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)
>> > >>>>>> ==28811==    by 0x41056E: main (ex56.c:395)
>> > >>>>>> ==28811==
>> > >>>>>> ==28812== Invalid read of size 16
>> > >>>>>> ==28812==    at 0x8550946: dswap_k_NEHALEM (in
>> > >>>>>> /usr/lib/openblas-base/libblas.so.3)
>> > >>>>>> ==28812==    by 0x7C6797F: dswap_ (in
>> /usr/lib/openblas-base/libblas
>> > >>>>>> .so.3)
>> > >>>>>> ==28812==    by 0x75B33B2: dgetri_ (in
>> /usr/lib/lapack/liblapack.so.3
>> > >>>>>> .0)
>> > >>>>>> ==28812==    by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012)
>> > >>>>>> ==28812==    by 0x5E320C9: PetscFESetUp (dtfe.c:3274)
>> > >>>>>> ==28812==    by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)
>> > >>>>>> ==28812==    by 0x41056E: main (ex56.c:395)
>> > >>>>>> ==28812==  Address 0xd9c7600 is 52,480 bytes inside a block of
>> size
>> > >>>>>> 52,488 alloc'd
>> > >>>>>> ==28812==    at 0x4C2D110: memalign (in
>> /usr/lib/valgrind/vgpreload_me
>> > >>>>>> mcheck-amd64-linux.so)
>> > >>>>>> ==28812==    by 0x51590F6: PetscMallocAlign (mal.c:39)
>> > >>>>>> ==28812==    by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983)
>> > >>>>>> ==28812==    by 0x5E320C9: PetscFESetUp (dtfe.c:3274)
>> > >>>>>> ==28812==    by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)
>> > >>>>>> ==28812==    by 0x41056E: main (ex56.c:395)
>> > >>>>>> ==28812==
>> > >>>>>> ==28811== Invalid read of size 16
>> > >>>>>> ==28811==    at 0x8550A55: dswap_k_NEHALEM (in
>> > >>>>>> /usr/lib/openblas-base/libblas.so.3)
>> > >>>>>> ==28811==    by 0x7C6797F: dswap_ (in
>> /usr/lib/openblas-base/libblas
>> > >>>>>> .so.3)
>> > >>>>>> ==28811==    by 0x7675179: dsteqr_ (in
>> /usr/lib/lapack/liblapack.so.3
>> > >>>>>> .0)
>> > >>>>>> ==28811==    by 0x5DFFA22: PetscDTGaussQuadrature (dt.c:508)
>> > >>>>>> ==28811==    by 0x5E00BD8: PetscDTGaussTensorQuadrature
>> (dt.c:582)
>> > >>>>>> ==28811==    by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)
>> > >>>>>> ==28811==    by 0x41056E: main (ex56.c:395)
>> > >>>>>> ==28811==  Address 0xd99cbe0 is 64 bytes inside a block of size
>> 72
>> > >>>>>> alloc'd
>> > >>>>>> ==28811==    at 0x4C2D110: memalign (in
>> /usr/lib/valgrind/vgpreload_me
>> > >>>>>> mcheck-amd64-linux.so)
>> > >>>>>> ==28811==    by 0x51590F6: PetscMallocAlign (mal.c:39)
>> > >>>>>> ==28811==    by 0x5DFF766: PetscDTGaussQuadrature (dt.c:504)
>> > >>>>>> ==28811==    by 0x5E00BD8: PetscDTGaussTensorQuadrature
>> (dt.c:582)
>> > >>>>>> ==28811==    by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)
>> > >>>>>> ==28811==    by 0x41056E: main (ex56.c:395)
>> > >>>>>> ==28811==
>> > >>>>>> ==28812== Invalid read of size 16
>> > >>>>>> ==28812==    at 0x8550A55: dswap_k_NEHALEM (in
>> > >>>>>> /usr/lib/openblas-base/libblas.so.3)
>> > >>>>>> ==28812==    by 0x7C6797F: dswap_ (in
>> /usr/lib/openblas-base/libblas
>> > >>>>>> .so.3)
>> > >>>>>> ==28812==    by 0x7675179: dsteqr_ (in
>> /usr/lib/lapack/liblapack.so.3
>> > >>>>>> .0)
>> > >>>>>> ==28812==    by 0x5DFFA22: PetscDTGaussQuadrature (dt.c:508)
>> > >>>>>> ==28812==    by 0x5E00BD8: PetscDTGaussTensorQuadrature
>> (dt.c:582)
>> > >>>>>> ==28812==    by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)
>> > >>>>>> ==28812==    by 0x41056E: main (ex56.c:395)
>> > >>>>>> ==28812==  Address 0xdc11f30 is 64 bytes inside a block of size
>> 72
>> > >>>>>> alloc'd
>> > >>>>>> ==28812==    at 0x4C2D110: memalign (in
>> /usr/lib/valgrind/vgpreload_me
>> > >>>>>> mcheck-amd64-linux.so)
>> > >>>>>> ==28812==    by 0x51590F6: PetscMallocAlign (mal.c:39)
>> > >>>>>> ==28812==    by 0x5DFF766: PetscDTGaussQuadrature (dt.c:504)
>> > >>>>>> ==28812==    by 0x5E00BD8: PetscDTGaussTensorQuadrature
>> (dt.c:582)
>> > >>>>>> ==28812==    by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)
>> > >>>>>> ==28812==    by 0x41056E: main (ex56.c:395)
>> > >>>>>> ==28812==
>> > >>>>>> [0] 27 global equations, 9 vertices
>> > >>>>>> [0] 27 equations in vector, 9 vertices
>> > >>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>> iterations 1
>> > >>>>>> [0] 441 global equations, 147 vertices
>> > >>>>>> [0] 441 equations in vector, 147 vertices
>> > >>>>>>
>> > >>>>>> hangs here ...
>> > >>>>>>
>> > >>>>>> Hong
>> > >>>>>>
>> > >>>>>> On Fri, Nov 10, 2017 at 8:57 AM, Mark Adams <mfadams at lbl.gov>
>> wrote:
>> > >>>>>>
>> > >>>>>>> This printed a little funny in gmail, snes/ex56 is running
>> clean in
>> > >>>>>>> the first few loops (appended), but the last one is the one
>> with a reduced
>> > >>>>>>> processor set. Still waiting. This is with 32 bit integers. I'm
>> running
>> > >>>>>>> another with 64 bit integers.
>> > >>>>>>>
>> > >>>>>>>    ...
>> > >>>>>>> [0] 27 global equations, 9 vertices
>> > >>>>>>> [0] 27 equations in vector, 9 vertices
>> > >>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>> iterations
>> > >>>>>>> 1
>> > >>>>>>> [0] 441 global equations, 147 vertices
>> > >>>>>>> [0] 441 equations in vector, 147 vertices
>> > >>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>> iterations
>> > >>>>>>> 1
>> > >>>>>>> [0] 4725 global equations, 1575 vertices
>> > >>>>>>> [0] 4725 equations in vector, 1575 vertices
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>> On Fri, Nov 10, 2017 at 9:06 AM, Mark Adams <mfadams at lbl.gov>
>> wrote:
>> > >>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> On Thu, Nov 9, 2017 at 1:56 PM, Hong <hzhang at mcs.anl.gov>
>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Mark:
>> > >>>>>>>>>
>> > >>>>>>>>>> OK, well, just go with the Linux machine for the regression
>> test.
>> > >>>>>>>>>> I will keep trying to reproduce this on my Mac with an O
>> build.
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> Valgrind error occurs on linux machines with g-build. I cannot
>> > >>>>>>>>> merge this branch to maint until the bug is fixed.
>> > >>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> Valgrind is failing on this run on my Mac. Moving to cg, like
>> you I
>> > >>>>>>>> suppose. This takes forever. This is what I have so far. Did
>> you get this
>> > >>>>>>>> far?
>> > >>>>>>>>
>> > >>>>>>>> 07:48 hzhang/fix-submat_samerowdist *=
>> > >>>>>>>> /sandbox/adams/petsc/src/snes/examples/tutorials$ make
>> > >>>>>>>> PETSC_DIR=/sandbox/adams/petsc PETSC_ARCH=arch-linux2-c-dbg32
>> val
>> > >>>>>>>> /sandbox/adams/petsc/arch-linux2-c-dbg32/bin/mpiexec -n 2
>> valgrind
>> > >>>>>>>> ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2
>> -snes_max_it 2
>> > >>>>>>>> -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type
>> > >>>>>>>> unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type
>> agg
>> > >>>>>>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>> > >>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>> > >>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
>> -snes_converged_reason
>> > >>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1
>> -mg_levels_ksp_type
>> > >>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg
>> -mg_levels_esteig_ksp_max_it 10
>> > >>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
>> -mg_levels_pc_type
>> > >>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size
>> 3 -run_type 1
>> > >>>>>>>> ==12414== Memcheck, a memory error detector
>> > >>>>>>>> ==12414== Copyright (C) 2002-2013, and GNU GPL'd, by Julian
>> Seward
>> > >>>>>>>> et al.
>> > >>>>>>>> ==12414== Using Valgrind-3.10.1 and LibVEX; rerun with -h for
>> > >>>>>>>> copyright info
>> > >>>>>>>> ==12415== Memcheck, a memory error detector
>> > >>>>>>>> ==12415== Copyright (C) 2002-2013, and GNU GPL'd, by Julian
>> Seward
>> > >>>>>>>> et al.
>> > >>>>>>>> ==12415== Using Valgrind-3.10.1 and LibVEX; rerun with -h for
>> > >>>>>>>> copyright info
>> > >>>>>>>> ==12415== Command: ./ex56 -cells 2,2,1 -max_conv_its 3
>> > >>>>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type
>> cg -ksp_rtol
>> > >>>>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10
>> -pc_type gamg
>> > >>>>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1
>> -pc_gamg_coarse_eq_limit 10
>> > >>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>> > >>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
>> -snes_converged_reason
>> > >>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1
>> -mg_levels_ksp_type
>> > >>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg
>> -mg_levels_esteig_ksp_max_it 10
>> > >>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
>> -mg_levels_pc_type
>> > >>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size
>> 3 -run_type 1
>> > >>>>>>>> ==12415==
>> > >>>>>>>> ==12414== Command: ./ex56 -cells 2,2,1 -max_conv_its 3
>> > >>>>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type
>> cg -ksp_rtol
>> > >>>>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10
>> -pc_type gamg
>> > >>>>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1
>> -pc_gamg_coarse_eq_limit 10
>> > >>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>> > >>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
>> -snes_converged_reason
>> > >>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1
>> -mg_levels_ksp_type
>> > >>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg
>> -mg_levels_esteig_ksp_max_it 10
>> > >>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
>> -mg_levels_pc_type
>> > >>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size
>> 3 -run_type 1
>> > >>>>>>>> ==12414==
>> > >>>>>>>> [0] 27 global equations, 9 vertices
>> > >>>>>>>> [0] 27 equations in vector, 9 vertices
>> > >>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>> > >>>>>>>> iterations 1
>> > >>>>>>>> [0] 441 global equations, 147 vertices
>> > >>>>>>>> [0] 441 equations in vector, 147 vertices
>> > >>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>> > >>>>>>>> iterations 1
>> > >>>>>>>> [0] 4725 global equations, 1575 vertices
>> > >>>>>>>> [0] 4725 equations in vector, 1575 vertices
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>>
>> > >>>>>
>> > >>>>
>> > >>>
>> > >>
>> > >
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171110/9313623b/attachment-0001.html>