[petsc-dev] Fwd: no petsc on Edison

[Hong]

Mark:
A quick check, I find that MatPtAP_MPIAIJ_MPIAI() set default algorithm as
rap=PETSC_TRUE; /* do R=P^T locally, then C=R*A*P */

The crash occurs at
3404]PETSC ERROR: #1 MatMatMultNumeric_SeqAIJ_SeqAIJ() line 187 in
/global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/seq/matmatmult.c

 ierr = PetscCalloc1(B->cmap->N,&ab_dense);

which is global column size P->N.
I'll check if we can avoid this non-scalable approach.
Meanwhile, can you try option '-matrap false' to use P^T*A*P?

Hong

We are getting this error on Edison. This problem has run before. This is
> using maint from last week.
>
> I recall there are low memory options for mpiptap. I don't think we are
> using them.
>
> Thanks,
>
> 3404]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [3404]PETSC ERROR: Out of memory. This could be due to allocating
> [3404]PETSC ERROR: too large an object or bleeding by not properly
> [3404]PETSC ERROR: destroying unneeded objects.
> [3404]PETSC ERROR: Memory allocated 0 Memory used by process 2537885696
> <(253)%20788-5696>
> [3404]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [3404]PETSC ERROR: Memory requested 657532760
> [3404]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [3404]PETSC ERROR: Petsc Release Version 3.7.5, unknown
> [3404]PETSC ERROR: /tmp/targetExec on a arch-xc30-opt64-intel named
> nid00713 by u6338 Wed Feb  1 16:15:09 2017
> [3404]PETSC ERROR: Configure options --COPTFLAGS="-fast -no-ipo"
> --CXXOPTFLAGS="-fast -no-ipo" --FOPTFLAGS="-fast -no-ipo"
> --download-parmetis --download-metis --download-hypre --with-ssl=0
> --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
> --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn
> --with-fortranlib-autodetect=0 --with-shared-libraries=0 --with-x=0
> --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices
> --prefix=/global/homes/m/madams/petsc_install/petsc-xc30-opt64-intel
> PETSC_ARCH=arch-xc30-opt64-intel PETSC_DIR=/global/homes/m/mada
> ms/petsc_install/petsc
> [3404]PETSC ERROR: #1 MatMatMultNumeric_SeqAIJ_SeqAIJ() line 187 in
> /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/seq/matmatmult.c
> [3404]PETSC ERROR: #2 MatMatMultNumeric_SeqAIJ_SeqAIJ() line 187 in
> /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/seq/matmatmult.c
> [3404]PETSC ERROR: #3 MatPtAPNumeric_MPIAIJ_MPIAIJ() line 557 in
> /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/mpi/mpiptap.c
> [3404]PETSC ERROR: #4 MatPtAP_MPIAIJ_MPIAIJ() line 110 in
> /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/mpi/mpiptap.c
> [3404]PETSC ERROR: #5 MatPtAP() line 9150 in /global/u2/m/madams/petsc_inst
> all/petsc/src/mat/interface/matrix.c
> [3404]PETSC ERROR: #6 PCGAMGCreateLevel_GAMG() line 88 in
> /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/gamg.c
> [3404]PETSC ERROR: #7 PCSetUp_GAMG() line 611 in
> /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/gamg.c
> [3404]PETSC ERROR: #8 PCSetUp() line 968 in /global/u2/m/madams/petsc_inst
> all/petsc/src/ksp/pc/interface/precon.c
> [3404]PETSC ERROR: #9 KSPSetUp() line 390 in /global/u2/m/madams/petsc_inst
> all/petsc/src/ksp/ksp/interface/itfunc.c
> [3404]PETSC ERROR: #10 KSPSolve() line 599 in
> /global/u2/m/madams/petsc_install/petsc/src/ksp/ksp/interface/itfunc.c
>
> [3406]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [3406]PETSC ERROR: Out of memory. This could be due to allocating
> [3406]PETSC ERROR: too large an object or bleeding by not properly
> [3406]PETSC ERROR: destroying unneeded objects.
> [3406]PETSC ERROR: Memory allocated 0 Memory used by process 2603798528
> <(260)%20379-8528>
> [3406]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [3406]PETSC ERROR: Memory requested 657532760
> [3406]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [3406]PETSC ERROR: Petsc Release Version 3.7.5, unknown
> [3406]PETSC ERROR: /tmp/targetExec on a arch-xc30-opt64-intel named
> nid00713 by u6338 Wed Feb  1 16:15:09 2017
> [3406]PETSC ERROR: Configure options --COPTFLAGS="-fast -no-ipo"
> --CXXOPTFLAGS="-fast -no-ipo" --FOPTFLAGS="-fast -no-ipo"
> --download-parmetis --download-metis --download-hypre --with-ssl=0
> --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
> --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn
> --with-fortranlib-autodetect=0 --with-shared-libraries=0 --with-x=0
> --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices
> --prefix=/global/homes/m/madams/petsc_install/petsc-xc30-opt64-intel
> PETSC_ARCH=arch-xc30-opt64-intel PETSC_DIR=/global/homes/m/madams/petsc_[7940]PETSC
> ERROR: -
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170202/d2749153/attachment.html>