[petsc-dev] Fwd: no petsc on Edison

Thu Feb 2 10:13:39 CST 2017

Mark:
Try '-matmatmult_via scalable' first. If this works, should we set it as
default?
Hong

>
>
> On Thu, Feb 2, 2017 at 10:53 AM, Satish Balay <balay at mcs.anl.gov> wrote:
>
>> > [3404]PETSC ERROR: Memory allocated 0 Memory used by process 2537885696
>> > [3404]PETSC ERROR: Memory requested 657532760
>>
>> So the process is currently using 2.5GB - and then made a malloc
>> request for 650MB - and this failed?
>>
>
> Yep, large but not garbage.
>
>
>>
>> How many 'tasks-per-node' are you using? Perhaps you need to reduce this
>> number?
>>
>
> Treb has run this job before and I think this code has been touched
> recently ... I suggested that Treb try -matmatmult_via scalable
>
> Thanks,
>
>
>>
>> https://www.nersc.gov/users/computational-systems/edison/configuration/
>> 64GB/node
>> <https://www.nersc.gov/users/computational-systems/edison/configuration/64GB/node>
>> => 5.3GB/process for tasks-per-node=12 [and 2.6GB/node for
>> tasks-per-node=24]
>>
>> Satish
>>
>> On Wed, 1 Feb 2017, Mark Adams wrote:
>>
>> > We are getting this error on Edison. This problem has run before. This
>> is
>> > using maint from last week.
>> >
>> > I recall there are low memory options for mpiptap. I don't think we are
>> > using them.
>> >
>> > Thanks,
>> >
>> > 3404]PETSC ERROR: --------------------- Error Message
>> > --------------------------------------------------------------
>> > [3404]PETSC ERROR: Out of memory. This could be due to allocating
>> > [3404]PETSC ERROR: too large an object or bleeding by not properly
>> > [3404]PETSC ERROR: destroying unneeded objects.
>> > [3404]PETSC ERROR: Memory allocated 0 Memory used by process 2537885696
>> > <(253)%20788-5696>
>>
>> > [3404]PETSC ERROR: Try running with -malloc_dump or -malloc_log for
>> info.
>> > [3404]PETSC ERROR: Memory requested 657532760
>> > [3404]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d
>> ocumentation/faq.html
>> > for trouble shooting.
>> > [3404]PETSC ERROR: Petsc Release Version 3.7.5, unknown
>> > [3404]PETSC ERROR: /tmp/targetExec on a arch-xc30-opt64-intel named
>> > nid00713 by u6338 Wed Feb  1 16:15:09 2017
>> > [3404]PETSC ERROR: Configure options --COPTFLAGS="-fast -no-ipo"
>> > --CXXOPTFLAGS="-fast -no-ipo" --FOPTFLAGS="-fast -no-ipo"
>> > --download-parmetis --download-metis --download-hypre --with-ssl=0
>> > --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
>> > --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn
>> > --with-fortranlib-autodetect=0 --with-shared-libraries=0 --with-x=0
>> > --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices
>> > --prefix=/global/homes/m/madams/petsc_install/petsc-xc30-opt64-intel
>> > PETSC_ARCH=arch-xc30-opt64-intel PETSC_DIR=/global/homes/m/mada
>> > ms/petsc_install/petsc
>> > [3404]PETSC ERROR: #1 MatMatMultNumeric_SeqAIJ_SeqAIJ() line 187 in
>> > /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/
>> seq/matmatmult.c
>> > [3404]PETSC ERROR: #2 MatMatMultNumeric_SeqAIJ_SeqAIJ() line 187 in
>> > /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/
>> seq/matmatmult.c
>> > [3404]PETSC ERROR: #3 MatPtAPNumeric_MPIAIJ_MPIAIJ() line 557 in
>> > /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/mpi/mpiptap.c
>> > [3404]PETSC ERROR: #4 MatPtAP_MPIAIJ_MPIAIJ() line 110 in
>> > /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/mpi/mpiptap.c
>> > [3404]PETSC ERROR: #5 MatPtAP() line 9150 in
>> /global/u2/m/madams/petsc_inst
>> > all/petsc/src/mat/interface/matrix.c
>> > [3404]PETSC ERROR: #6 PCGAMGCreateLevel_GAMG() line 88 in
>> > /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/gamg.c
>> > [3404]PETSC ERROR: #7 PCSetUp_GAMG() line 611 in
>> > /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/gamg.c
>> > [3404]PETSC ERROR: #8 PCSetUp() line 968 in
>> /global/u2/m/madams/petsc_inst
>> > all/petsc/src/ksp/pc/interface/precon.c
>> > [3404]PETSC ERROR: #9 KSPSetUp() line 390 in
>> /global/u2/m/madams/petsc_inst
>> > all/petsc/src/ksp/ksp/interface/itfunc.c
>> > [3404]PETSC ERROR: #10 KSPSolve() line 599 in
>> /global/u2/m/madams/petsc_inst
>> > all/petsc/src/ksp/ksp/interface/itfunc.c
>> >
>> > [3406]PETSC ERROR: --------------------- Error Message
>> > --------------------------------------------------------------
>> > [3406]PETSC ERROR: Out of memory. This could be due to allocating
>> > [3406]PETSC ERROR: too large an object or bleeding by not properly
>> > [3406]PETSC ERROR: destroying unneeded objects.
>> > [3406]PETSC ERROR: Memory allocated 0 Memory used by process 2603798528
>> <(260)%20379-8528>
>> > <(260)%20379-8528>
>> > [3406]PETSC ERROR: Try running with -malloc_dump or -malloc_log for
>> info.
>> > [3406]PETSC ERROR: Memory requested 657532760
>> > [3406]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d
>> ocumentation/faq.html
>> > for trouble shooting.
>> > [3406]PETSC ERROR: Petsc Release Version 3.7.5, unknown
>> > [3406]PETSC ERROR: /tmp/targetExec on a arch-xc30-opt64-intel named
>> > nid00713 by u6338 Wed Feb  1 16:15:09 2017
>> > [3406]PETSC ERROR: Configure options --COPTFLAGS="-fast -no-ipo"
>> > --CXXOPTFLAGS="-fast -no-ipo" --FOPTFLAGS="-fast -no-ipo"
>> > --download-parmetis --download-metis --download-hypre --with-ssl=0
>> > --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
>> > --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn
>> > --with-fortranlib-autodetect=0 --with-shared-libraries=0 --with-x=0
>> > --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices
>> > --prefix=/global/homes/m/madams/petsc_install/petsc-xc30-opt64-intel
>> > PETSC_ARCH=arch-xc30-opt64-intel
>> > PETSC_DIR=/global/homes/m/madams/petsc_[7940]PETSC
>> > ERROR: -
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170202/8c5afeee/attachment.html>