memory check of /snes/example/tutorials/ex29.c

Barry Smith bsmith at mcs.anl.gov
Mon Jul 27 20:35:37 CDT 2009 

     The memory for the matrix will be (321*321*4) * (25*  
4)                              *  
(12)                                           = 494,596,800  which is  
500 megabytes
                                                                   ( #  
rows)         (nonzeros per row)      (1 double + 1 int per nonzero)

There are 25 * 4 nonzeros per row because you have box stencil of  
width 2.  DAGetMatrix() has no way of knowing that your equations have  
only 13 nonzeros per row it has to assume a nonzero for each possible  
couple in the stencil.  You can use DASetBlockFills() to indicate  
which parts of the stencil are truly nonzero and thus greatly reduce  
the matrix memory usage.

    Barry


On Jul 27, 2009, at 5:04 PM, (Rebecca) Xuefei YUAN wrote:

> Dear Matt,
>
> I ran the code on a 321*321 grid, with dof=4. The matrix is a sparse  
> matrix with type aij.
>
> After set up user defined options calls, the memory status is
> Mem:   2033752k total,   607456k used,  1426296k free,     4832k  
> buffers
>
>        ierr = DMMGCreate(comm, parameters.numberOfLevels, &appCtx,  
> &dmmg);CHKERRQ(ierr);
> 	ierr = DACreate2d(comm,DA_NONPERIODIC,DA_STENCIL_BOX, -5, -5,  
> PETSC_DECIDE, PETSC_DECIDE, 4, 2, 0, 0, &da);CHKERRQ(ierr);
> 	ierr = DMMGSetDM(dmmg, (DM)da);CHKERRQ(ierr);
> 	ierr = DASetFieldName(DMMGGetDA(dmmg), 0, "phi");CHKERRQ(ierr);
> 	ierr = DASetFieldName(DMMGGetDA(dmmg), 1, "vz");CHKERRQ(ierr);
> 	ierr = DASetFieldName(DMMGGetDA(dmmg), 2, "psi");CHKERRQ(ierr);
> 	ierr = DASetFieldName(DMMGGetDA(dmmg), 3, "bz");CHKERRQ(ierr);
>
>
> before DAGetMatrix() called, the memory status is
> Mem:   2033752k total,   642940k used,  1390812k free,     4972k  
> buffers
>
> 	ierr = DAGetMatrix(DMMGGetDA(dmmg), MATAIJ, &jacobian);CHKERRQ(ierr);
>
> In gdb, it uses around 500M memory after DAGetMatrix(), which I do  
> not think it is right, since for a sparse matrix with 13 nonzeros  
> per row, the memory it needs should be 321*321*4(dof)*13(nonzeros  
> per row)*8(PetscReal) = 42865056 bytes ~ 40M. It is strange.
> i.e., after DAGetMatrix() call, the memory status is
>
> Mem:   2033752k total,  1152032k used,   881720k free,     5072k  
> buffers
>
> Then when it goes the call of DMMGSetSNESLocal(), I found my memory  
> is using till the message of corruption has appeared.
>
> 	ierr = DMMGSetSNESLocal(dmmg, FormFunctionLocal, FormJacobianLocal, 
> 0,0);CHKERRQ(ierr);
>
> The memory corruption happens. The error message is:
>
>  0 SNES Function norm 4.925849247379e-03
> [0]PETSC ERROR: --------------------- Error Message  
> ------------------------------------
> [0]PETSC ERROR: Out of memory. This could be due to allocating
> [0]PETSC ERROR: too large an object or bleeding by not properly
> [0]PETSC ERROR: destroying unneeded objects.
> [0]PETSC ERROR: Memory allocated 2134287012 Memory used by process  
> 1630638080
> [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [0]PETSC ERROR: Memory requested 327270824!
> [0]PETSC ERROR:  
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 1, Thu Jan  1  
> 13:54:27 CST 2009
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:  
> ------------------------------------------------------------------------
> [0]PETSC ERROR: /home/rebecca/linux/code/physics/qffxmhd/tests/ 
> qffxmhd3 on a linux-gnu named YuanWork by rebecca Mon Jul 27  
> 17:49:53 2009
> [0]PETSC ERROR: Libraries linked from /home/rebecca/soft/petsc-3.0.0- 
> p1/linux-gnu-c-debug/lib
> [0]PETSC ERROR: Configure run at Mon Apr 20 16:41:56 2009
> [0]PETSC ERROR: Configure options --with-blas-lapack-dir=./ 
> externalpackages/fblaslapack-3.1.1/ --download-mpich=1 --with-shared=0
> [0]PETSC ERROR:  
> ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscMallocAlign() line 61 in src/sys/memory/mal.c
> [0]PETSC ERROR: PetscTrMallocDefault() line 194 in src/sys/memory/ 
> mtr.c
> [0]PETSC ERROR: MatDuplicateNoCreate_SeqAIJ() line 3402 in src/mat/ 
> impls/aij/seq/aij.c
> [0]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1241 in src/mat/ 
> impls/aij/seq/aijfact.c
> [0]PETSC ERROR: MatILUFactorSymbolic() line 5243 in src/mat/ 
> interface/matrix.c
> [0]PETSC ERROR: PCSetUp_ILU() line 293 in src/ksp/pc/impls/factor/ 
> ilu/ilu.c
> [0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: PCSetUp_MG() line 516 in src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: SNES_KSPSolve() line 2899 in src/snes/interface/snes.c
> [0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c
> [0]PETSC ERROR: SNESSolve() line 2221 in src/snes/interface/snes.c
> [0]PETSC ERROR: DMMGSolveSNES() line 510 in src/snes/utils/damgsnes.c
> [0]PETSC ERROR: DMMGSolve() line 372 in src/snes/utils/damg.c
> [0]PETSC ERROR: Solve() line 318 in qffxmhd.c
> [0]PETSC ERROR: main() line 172 in qffxmhd.c
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0[unset]:  
> aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
>
> Program exited with code 01.
>
>
>  0 SNES Function norm 4.925849247379e-03
> [0]PETSC ERROR: --------------------- Error Message  
> ------------------------------------
> [0]PETSC ERROR: Out of memory. This could be due to allocating
> [0]PETSC ERROR: too large an object or bleeding by not properly
> [0]PETSC ERROR: destroying unneeded objects.
> [0]PETSC ERROR: Memory allocated 2134287012 Memory used by process  
> 1630638080
> [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [0]PETSC ERROR: Memory requested 327270824!
> [0]PETSC ERROR:  
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 1, Thu Jan  1  
> 13:54:27 CST 2009
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:  
> ------------------------------------------------------------------------
> [0]PETSC ERROR: /home/rebecca/linux/code/physics/qffxmhd/tests/ 
> qffxmhd3 on a linux-gnu named YuanWork by rebecca Mon Jul 27  
> 17:49:53 2009
> [0]PETSC ERROR: Libraries linked from /home/rebecca/soft/petsc-3.0.0- 
> p1/linux-gnu-c-debug/lib
> [0]PETSC ERROR: Configure run at Mon Apr 20 16:41:56 2009
> [0]PETSC ERROR: Configure options --with-blas-lapack-dir=./ 
> externalpackages/fblaslapack-3.1.1/ --download-mpich=1 --with-shared=0
> [0]PETSC ERROR:  
> ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscMallocAlign() line 61 in src/sys/memory/mal.c
> [0]PETSC ERROR: PetscTrMallocDefault() line 194 in src/sys/memory/ 
> mtr.c
> [0]PETSC ERROR: MatDuplicateNoCreate_SeqAIJ() line 3402 in src/mat/ 
> impls/aij/seq/aij.c
> [0]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1241 in src/mat/ 
> impls/aij/seq/aijfact.c
> [0]PETSC ERROR: MatILUFactorSymbolic() line 5243 in src/mat/ 
> interface/matrix.c
> [0]PETSC ERROR: PCSetUp_ILU() line 293 in src/ksp/pc/impls/factor/ 
> ilu/ilu.c
> [0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: PCSetUp_MG() line 516 in src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: SNES_KSPSolve() line 2899 in src/snes/interface/snes.c
> [0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c
> [0]PETSC ERROR: SNESSolve() line 2221 in src/snes/interface/snes.c
> [0]PETSC ERROR: DMMGSolveSNES() line 510 in src/snes/utils/damgsnes.c
> [0]PETSC ERROR: DMMGSolve() line 372 in src/snes/utils/damg.c
> [0]PETSC ERROR: Solve() line 318 in qffxmhd.c
> [0]PETSC ERROR: main() line 172 in qffxmhd.c
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0[unset]:  
> aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
>
> Program exited with code 01.
>
> I thought it might because of the unfreed memory, so I picked up  
> ex29.c as a comparision.
>
> Thanks,
>
> Rebecca
>
>
> Quoting Matthew Knepley <knepley at gmail.com>:
>
>> On Mon, Jul 27, 2009 at 4:34 PM, (Rebecca) Xuefei YUAN
>> <xy2102 at columbia.edu>wrote:
>>
>>> Those unfreed bytes cause "out of memory" when it runs at bigger  
>>> grid
>>> sizes. So I have to find out those unfreed memory and free them...  
>>> Any
>>> suggestions?
>>
>>
>> Not from what you mailed in. On that DA line, I see  
>> PetscHeaderCreate(). Is
>> that what you see?
>>
>>  Matt
>>
>>
>>>
>>> Thanks,
>>>
>>> R
>>>
>>>
>>> Quoting Matthew Knepley <knepley at gmail.com>:
>>>
>>> On Mon, Jul 27, 2009 at 3:51 PM, (Rebecca) Xuefei YUAN
>>>> <xy2102 at columbia.edu>wrote:
>>>>
>>>> Hi,
>>>>>
>>>>> My own code has some left bytes still reachable according to  
>>>>> valgrind,
>>>>> then
>>>>> I use two different version petsc (2.3.3-p15 and 3.0.0-p1) to  
>>>>> compile and
>>>>> make the files, it gives me different number of bytes left still
>>>>> reachable.
>>>>> Moreover, I picked up the /snes/example/tutorials/ex29.c as  
>>>>> another
>>>>> example,
>>>>> and found that some bytes are still reachable, what is the cause  
>>>>> of it?
>>>>> It
>>>>> shows that it is from DACreate2D() and the I use -malloc_dump to  
>>>>> get
>>>>> those
>>>>> unfreed informations.
>>>>>
>>>>> I understand that for those 5 loss record, the 2nd, 3rd and 4th  
>>>>> are true
>>>>> for all examples, but where do 1st and 5th ones come from? Also,  
>>>>> the
>>>>> -malloc_dump information shows that there are
>>>>> "[0]Total space allocated 37780 bytes",
>>>>> but valgrind gives the information as
>>>>> "==26628==    still reachable: 132,828 bytes in 323 blocks"
>>>>>
>>>>> Why there is a big difference?
>>>>>
>>>>
>>>>
>>>> 1 is fine. It is from PMPI setup, which has some bytes not freed  
>>>> from
>>>> setting up the MPI
>>>> processes. The last one looks like an unfreed header for a DA,  
>>>> which is
>>>> strange.
>>>>
>>>> Matt
>>>>
>>>>
>>>>
>>>>> Thanks very much!
>>>>>
>>>>> Rebecca
>>>>>
>>>>> Here is the message from valgrind of running ex29:
>>>>> ==26628== 32 bytes in 2 blocks are still reachable in loss  
>>>>> record 1 of 5
>>>>> ==26628==    at 0x4022AB8: malloc (vg_replace_malloc.c:207)
>>>>> ==26628==    by 0x86F9A78: MPID_VCRT_Create (mpid_vc.c:62)
>>>>> ==26628==    by 0x86F743A: MPID_Init (mpid_init.c:116)
>>>>> ==26628==    by 0x86D040B: MPIR_Init_thread (initthread.c:288)
>>>>> ==26628==    by 0x86CFF2D: PMPI_Init (init.c:106)
>>>>> ==26628==    by 0x8613D69: PetscInitialize (pinit.c:503)
>>>>> ==26628==    by 0x804B796: main (ex29.c:139)
>>>>> ==26628==
>>>>> ==26628==
>>>>> ==26628== 156 (36 direct, 120 indirect) bytes in 1 blocks are  
>>>>> definitely
>>>>> lost in loss record 2 of 5
>>>>> ==26628==    at 0x4022AB8: malloc (vg_replace_malloc.c:207)
>>>>> ==26628==    by 0x429B3E2: (within /lib/tls/i686/cmov/libc-2.7.so)
>>>>> ==26628==    by 0x429BC2D: __nss_database_lookup (in /lib/tls/ 
>>>>> i686/cmov/
>>>>> libc-2.7.so)
>>>>> ==26628==    by 0x4732FDB: ???
>>>>> ==26628==    by 0x473413C: ???
>>>>> ==26628==    by 0x4247D15: getpwuid_r (in /lib/tls/i686/cmov/ 
>>>>> libc-2.7.so
>>>>> )
>>>>> ==26628==    by 0x424765D: getpwuid (in /lib/tls/i686/cmov/ 
>>>>> libc-2.7.so)
>>>>> ==26628==    by 0x8623509: PetscGetUserName (fuser.c:68)
>>>>> ==26628==    by 0x85E0CF0: PetscErrorPrintfInitialize  
>>>>> (errtrace.c:68)
>>>>> ==26628==    by 0x8613E23: PetscInitialize (pinit.c:518)
>>>>> ==26628==    by 0x804B796: main (ex29.c:139)
>>>>> ==26628==
>>>>> ==26628==
>>>>> ==26628== 40 bytes in 5 blocks are indirectly lost in loss  
>>>>> record 3 of 5
>>>>> ==26628==    at 0x4022AB8: malloc (vg_replace_malloc.c:207)
>>>>> ==26628==    by 0x429AFBB: __nss_lookup_function (in /lib/tls/ 
>>>>> i686/cmov/
>>>>> libc-2.7.so)
>>>>> ==26628==    by 0x4732FFB: ???
>>>>> ==26628==    by 0x473413C: ???
>>>>> ==26628==    by 0x4247D15: getpwuid_r (in /lib/tls/i686/cmov/ 
>>>>> libc-2.7.so
>>>>> )
>>>>> ==26628==    by 0x424765D: getpwuid (in /lib/tls/i686/cmov/ 
>>>>> libc-2.7.so)
>>>>> ==26628==    by 0x8623509: PetscGetUserName (fuser.c:68)
>>>>> ==26628==    by 0x85E0CF0: PetscErrorPrintfInitialize  
>>>>> (errtrace.c:68)
>>>>> ==26628==    by 0x8613E23: PetscInitialize (pinit.c:518)
>>>>> ==26628==    by 0x804B796: main (ex29.c:139)
>>>>> ==26628==
>>>>> ==26628==
>>>>> ==26628== 80 bytes in 5 blocks are indirectly lost in loss  
>>>>> record 4 of 5
>>>>> ==26628==    at 0x4022AB8: malloc (vg_replace_malloc.c:207)
>>>>> ==26628==    by 0x428839B: tsearch (in /lib/tls/i686/cmov/ 
>>>>> libc-2.7.so)
>>>>> ==26628==    by 0x429AF7D: __nss_lookup_function (in /lib/tls/ 
>>>>> i686/cmov/
>>>>> libc-2.7.so)
>>>>> ==26628==    by 0x4732FFB: ???
>>>>> ==26628==    by 0x473413C: ???
>>>>> ==26628==    by 0x4247D15: getpwuid_r (in /lib/tls/i686/cmov/ 
>>>>> libc-2.7.so
>>>>> )
>>>>> ==26628==    by 0x424765D: getpwuid (in /lib/tls/i686/cmov/ 
>>>>> libc-2.7.so)
>>>>> ==26628==    by 0x8623509: PetscGetUserName (fuser.c:68)
>>>>> ==26628==    by 0x85E0CF0: PetscErrorPrintfInitialize  
>>>>> (errtrace.c:68)
>>>>> ==26628==    by 0x8613E23: PetscInitialize (pinit.c:518)
>>>>> ==26628==    by 0x804B796: main (ex29.c:139)
>>>>> ==26628==
>>>>> ==26628==
>>>>> ==26628== 132,796 bytes in 321 blocks are still reachable in  
>>>>> loss record
>>>>> 5
>>>>> of 5
>>>>> ==26628==    at 0x4022AB8: malloc (vg_replace_malloc.c:207)
>>>>> ==26628==    by 0x85EF3AC: PetscMallocAlign (mal.c:40)
>>>>> ==26628==    by 0x85F049B: PetscTrMallocDefault (mtr.c:194)
>>>>> ==26628==    by 0x81BCD3F: DACreate2d (da2.c:364)
>>>>> ==26628==    by 0x804BAFB: main (ex29.c:153)
>>>>> ==26628==
>>>>> ==26628== LEAK SUMMARY:
>>>>> ==26628==    definitely lost: 36 bytes in 1 blocks.
>>>>> ==26628==    indirectly lost: 120 bytes in 10 blocks.
>>>>> ==26628==      possibly lost: 0 bytes in 0 blocks.
>>>>> ==26628==    still reachable: 132,828 bytes in 323 blocks.
>>>>> ==26628==         suppressed: 0 bytes in 0 blocks.
>>>>>
>>>>>
>>>>> --
>>>>> (Rebecca) Xuefei YUAN
>>>>> Department of Applied Physics and Applied Mathematics
>>>>> Columbia University
>>>>> Tel:917-399-8032
>>>>> www.columbia.edu/~xy2102 <http://www.columbia.edu/%7Exy2102> <
>>>>> http://www.columbia.edu/%7Exy2102>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments
>>>> is infinitely more interesting than any results to which their  
>>>> experiments
>>>> lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>
>>>
>>> --
>>> (Rebecca) Xuefei YUAN
>>> Department of Applied Physics and Applied Mathematics
>>> Columbia University
>>> Tel:917-399-8032
>>> www.columbia.edu/~xy2102 <http://www.columbia.edu/%7Exy2102>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their  
>> experiments
>> is infinitely more interesting than any results to which their  
>> experiments
>> lead.
>> -- Norbert Wiener
>>
>
>
>
> -- 
> (Rebecca) Xuefei YUAN
> Department of Applied Physics and Applied Mathematics
> Columbia University
> Tel:917-399-8032
> www.columbia.edu/~xy2102
>

More information about the petsc-users mailing list