[petsc-dev] Reduction in KSPGetVecs via DMCreateGlobalVector

[Barry Smith]

  At one point DMCreateGlobalVector_DA() did a duplicate of a “base” global vector, then in theory it could share the layout but that is all different now.

   Note that the basic VecDuplicate_MPI() does share the previous layout but then there is a VecDuplicate_MPI_DA() which does not share the layout.

   At some time some of this code got changed and whoever changed it didn’t worry enough about reusing the layout etc. 

   Yes should be fixed.

   Barry

On Mar 26, 2014, at 7:00 AM, Jed Brown <jed at jedbrown.org> wrote:

> I don't think it's reasonable to allow this:
> 
> #0  0x00007ffff5af9cc0 in PMPI_Allreduce () from /opt/mpich/lib/libmpich.so.12
> #1  0x00007ffff61b7436 in PetscSplitOwnership () from /home/jed/petsc/mpich-opt/lib/libpetscsys.so.3.04
> #2  0x00007ffff65a44b6 in PetscLayoutSetUp () from /home/jed/petsc/mpich-opt/lib/libpetscvec.so.3.04
> #3  0x00007ffff65ff988 in VecCreate_MPI_Private () from /home/jed/petsc/mpich-opt/lib/libpetscvec.so.3.04
> #4  0x00007ffff6600082 in VecCreate_MPI () from /home/jed/petsc/mpich-opt/lib/libpetscvec.so.3.04
> #5  0x00007ffff6610867 in VecSetType () from /home/jed/petsc/mpich-opt/lib/libpetscvec.so.3.04
> #6  0x00007ffff660010f in VecCreate_Standard () from /home/jed/petsc/mpich-opt/lib/libpetscvec.so.3.04
> #7  0x00007ffff6610867 in VecSetType () from /home/jed/petsc/mpich-opt/lib/libpetscvec.so.3.04
> #8  0x00007ffff6fbe98b in DMCreateGlobalVector_DA () from /home/jed/petsc/mpich-opt/lib/libpetscdm.so.3.04
> #9  0x00007ffff70d979e in DMCreateGlobalVector () from /home/jed/petsc/mpich-opt/lib/libpetscdm.so.3.04
> #10 0x00007ffff6fbe7d0 in VecDuplicate_MPI_DA () from /home/jed/petsc/mpich-opt/lib/libpetscdm.so.3.04
> #11 0x00007ffff660ca11 in VecDuplicate () from /home/jed/petsc/mpich-opt/lib/libpetscvec.so.3.04
> #12 0x00007ffff660cf37 in VecDuplicateVecs_Default () from /home/jed/petsc/mpich-opt/lib/libpetscvec.so.3.04
> #13 0x00007ffff660cc2e in VecDuplicateVecs () from /home/jed/petsc/mpich-opt/lib/libpetscvec.so.3.04
> #14 0x00007ffff7569da1 in KSPGetVecs () from /home/jed/petsc/mpich-opt/lib/libpetscksp.so.3.04
> #15 0x00007ffff753b403 in KSPGMRESGetNewVectors () from /home/jed/petsc/mpich-opt/lib/libpetscksp.so.3.04
> #16 0x00007ffff753b748 in KSPGMRESCycle () from /home/jed/petsc/mpich-opt/lib/libpetscksp.so.3.04
> #17 0x00007ffff753c0cc in KSPSolve_GMRES () from /home/jed/petsc/mpich-opt/lib/libpetscksp.so.3.04
> #18 0x00007ffff755e6b0 in KSPSolve () from /home/jed/petsc/mpich-opt/lib/libpetscksp.so.3.04
> #19 0x00007ffff7b9cdb0 in SNESSolve_NEWTONLS () from /home/jed/petsc/mpich-opt/lib/libpetscsnes.so.3.04
> #20 0x00007ffff7b73c85 in SNESSolve () from /home/jed/petsc/mpich-opt/lib/libpetscsnes.so.3.04
> #21 0x000000000040183a in main ()
> 
> Found by breaking on PMPI_Allreduce inside
> 
> ~/petsc/src/snes/examples/tutorials$ mpiexec -n 1 gdb --args ./ex5 -da_refine 3 -ksp_max_it 5 : -n 1 ./ex5 -da_refine 3 -ksp_max_it 5