On Mon, Jul 27, 2009 at 5:04 PM, (Rebecca) Xuefei YUAN <span dir="ltr"><<a href="mailto:xy2102@columbia.edu">xy2102@columbia.edu</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Dear Matt,<br>
<br>
I ran the code on a 321*321 grid, with dof=4. The matrix is a sparse matrix with type aij.<br>
<br>
After set up user defined options calls, the memory status is<br>
Mem: 2033752k total, 607456k used, 1426296k free, 4832k buffers<br>
<br>
ierr = DMMGCreate(comm, parameters.numberOfLevels, &appCtx, &dmmg);CHKERRQ(ierr);<br>
ierr = DACreate2d(comm,DA_NONPERIODIC,DA_STENCIL_BOX, -5, -5, PETSC_DECIDE, PETSC_DECIDE, 4, 2, 0, 0, &da);CHKERRQ(ierr);<br>
ierr = DMMGSetDM(dmmg, (DM)da);CHKERRQ(ierr);<br>
ierr = DASetFieldName(DMMGGetDA(dmmg), 0, "phi");CHKERRQ(ierr);<br>
ierr = DASetFieldName(DMMGGetDA(dmmg), 1, "vz");CHKERRQ(ierr);<br>
ierr = DASetFieldName(DMMGGetDA(dmmg), 2, "psi");CHKERRQ(ierr);<br>
ierr = DASetFieldName(DMMGGetDA(dmmg), 3, "bz");CHKERRQ(ierr);<br>
<br>
<br>
before DAGetMatrix() called, the memory status is<br>
Mem: 2033752k total, 642940k used, 1390812k free, 4972k buffers<br>
<br>
ierr = DAGetMatrix(DMMGGetDA(dmmg), MATAIJ, &jacobian);CHKERRQ(ierr);<br>
<br>
In gdb, it uses around 500M memory after DAGetMatrix(), which I do not think it is right, since for a sparse matrix with 13 nonzeros per row, the memory it needs should be 321*321*4(dof)*13(nonzeros per row)*8(PetscReal) = 42865056 bytes ~ 40M. It is strange.<br>
i.e., after DAGetMatrix() call, the memory status is</blockquote><div><br>Are you sure you have stencil width 1? This is not the only memory used by a matrix, but should be close. The<br>number of nonzero is reported by -ksp_view. Check this.<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
Mem: 2033752k total, 1152032k used, 881720k free, 5072k buffers<br>
<br>
Then when it goes the call of DMMGSetSNESLocal(), I found my memory is using till the message of corruption has appeared.<br>
<br>
ierr = DMMGSetSNESLocal(dmmg, FormFunctionLocal, FormJacobianLocal,0,0);CHKERRQ(ierr);<br>
<br>
The memory corruption happens. The error message is:</blockquote><div><br>This is not corruption, just using up memory.<br><br> Matt<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
0 SNES Function norm 4.925849247379e-03<br>
[0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
[0]PETSC ERROR: Out of memory. This could be due to allocating<br>
[0]PETSC ERROR: too large an object or bleeding by not properly<br>
[0]PETSC ERROR: destroying unneeded objects.<br>
[0]PETSC ERROR: Memory allocated 2134287012 Memory used by process 1630638080<br>
[0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.<br>
[0]PETSC ERROR: Memory requested 327270824!<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 1, Thu Jan 1 13:54:27 CST 2009<br>
[0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[0]PETSC ERROR: See docs/index.html for manual pages.<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: /home/rebecca/linux/code/physics/qffxmhd/tests/qffxmhd3 on a linux-gnu named YuanWork by rebecca Mon Jul 27 17:49:53 2009<br>
[0]PETSC ERROR: Libraries linked from /home/rebecca/soft/petsc-3.0.0-p1/linux-gnu-c-debug/lib<br>
[0]PETSC ERROR: Configure run at Mon Apr 20 16:41:56 2009<br>
[0]PETSC ERROR: Configure options --with-blas-lapack-dir=./externalpackages/fblaslapack-3.1.1/ --download-mpich=1 --with-shared=0<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: PetscMallocAlign() line 61 in src/sys/memory/mal.c<br>
[0]PETSC ERROR: PetscTrMallocDefault() line 194 in src/sys/memory/mtr.c<br>
[0]PETSC ERROR: MatDuplicateNoCreate_SeqAIJ() line 3402 in src/mat/impls/aij/seq/aij.c<br>
[0]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1241 in src/mat/impls/aij/seq/aijfact.c<br>
[0]PETSC ERROR: MatILUFactorSymbolic() line 5243 in src/mat/interface/matrix.c<br>
[0]PETSC ERROR: PCSetUp_ILU() line 293 in src/ksp/pc/impls/factor/ilu/ilu.c<br>
[0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c<br>
[0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c<br>
[0]PETSC ERROR: PCSetUp_MG() line 516 in src/ksp/pc/impls/mg/mg.c<br>
[0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c<br>
[0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c<br>
[0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c<br>
[0]PETSC ERROR: SNES_KSPSolve() line 2899 in src/snes/interface/snes.c<br>
[0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c<br>
[0]PETSC ERROR: SNESSolve() line 2221 in src/snes/interface/snes.c<br>
[0]PETSC ERROR: DMMGSolveSNES() line 510 in src/snes/utils/damgsnes.c<br>
[0]PETSC ERROR: DMMGSolve() line 372 in src/snes/utils/damg.c<br>
[0]PETSC ERROR: Solve() line 318 in qffxmhd.c<br>
[0]PETSC ERROR: main() line 172 in qffxmhd.c<br>
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0[unset]: aborting job:<br>
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0<br>
<br>
Program exited with code 01.<br>
<br>
<br>
0 SNES Function norm 4.925849247379e-03<br>
[0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
[0]PETSC ERROR: Out of memory. This could be due to allocating<br>
[0]PETSC ERROR: too large an object or bleeding by not properly<br>
[0]PETSC ERROR: destroying unneeded objects.<br>
[0]PETSC ERROR: Memory allocated 2134287012 Memory used by process 1630638080<br>
[0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.<br>
[0]PETSC ERROR: Memory requested 327270824!<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 1, Thu Jan 1 13:54:27 CST 2009<br>
[0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[0]PETSC ERROR: See docs/index.html for manual pages.<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: /home/rebecca/linux/code/physics/qffxmhd/tests/qffxmhd3 on a linux-gnu named YuanWork by rebecca Mon Jul 27 17:49:53 2009<br>
[0]PETSC ERROR: Libraries linked from /home/rebecca/soft/petsc-3.0.0-p1/linux-gnu-c-debug/lib<br>
[0]PETSC ERROR: Configure run at Mon Apr 20 16:41:56 2009<br>
[0]PETSC ERROR: Configure options --with-blas-lapack-dir=./externalpackages/fblaslapack-3.1.1/ --download-mpich=1 --with-shared=0<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: PetscMallocAlign() line 61 in src/sys/memory/mal.c<br>
[0]PETSC ERROR: PetscTrMallocDefault() line 194 in src/sys/memory/mtr.c<br>
[0]PETSC ERROR: MatDuplicateNoCreate_SeqAIJ() line 3402 in src/mat/impls/aij/seq/aij.c<br>
[0]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1241 in src/mat/impls/aij/seq/aijfact.c<br>
[0]PETSC ERROR: MatILUFactorSymbolic() line 5243 in src/mat/interface/matrix.c<br>
[0]PETSC ERROR: PCSetUp_ILU() line 293 in src/ksp/pc/impls/factor/ilu/ilu.c<br>
[0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c<br>
[0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c<br>
[0]PETSC ERROR: PCSetUp_MG() line 516 in src/ksp/pc/impls/mg/mg.c<br>
[0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c<br>
[0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c<br>
[0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c<br>
[0]PETSC ERROR: SNES_KSPSolve() line 2899 in src/snes/interface/snes.c<br>
[0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c<br>
[0]PETSC ERROR: SNESSolve() line 2221 in src/snes/interface/snes.c<br>
[0]PETSC ERROR: DMMGSolveSNES() line 510 in src/snes/utils/damgsnes.c<br>
[0]PETSC ERROR: DMMGSolve() line 372 in src/snes/utils/damg.c<br>
[0]PETSC ERROR: Solve() line 318 in qffxmhd.c<br>
[0]PETSC ERROR: main() line 172 in qffxmhd.c<br>
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0[unset]: aborting job:<br>
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0<br>
<br>
Program exited with code 01.<br>
<br>
I thought it might because of the unfreed memory, so I picked up ex29.c as a comparision.<br>
<br>
Thanks,<br><font color="#888888">
<br>
Rebecca</font><div><div></div><div class="h5"><br>
<br>
<br>
Quoting Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On Mon, Jul 27, 2009 at 4:34 PM, (Rebecca) Xuefei YUAN<br>
<<a href="mailto:xy2102@columbia.edu" target="_blank">xy2102@columbia.edu</a>>wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Those unfreed bytes cause "out of memory" when it runs at bigger grid<br>
sizes. So I have to find out those unfreed memory and free them... Any<br>
suggestions?<br>
</blockquote>
<br>
<br>
Not from what you mailed in. On that DA line, I see PetscHeaderCreate(). Is<br>
that what you see?<br>
<br>
Matt<br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
Thanks,<br>
<br>
R<br>
<br>
<br>
Quoting Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>:<br>
<br>
On Mon, Jul 27, 2009 at 3:51 PM, (Rebecca) Xuefei YUAN<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<<a href="mailto:xy2102@columbia.edu" target="_blank">xy2102@columbia.edu</a>>wrote:<br>
<br>
Hi,<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
My own code has some left bytes still reachable according to valgrind,<br>
then<br>
I use two different version petsc (2.3.3-p15 and 3.0.0-p1) to compile and<br>
make the files, it gives me different number of bytes left still<br>
reachable.<br>
Moreover, I picked up the /snes/example/tutorials/ex29.c as another<br>
example,<br>
and found that some bytes are still reachable, what is the cause of it?<br>
It<br>
shows that it is from DACreate2D() and the I use -malloc_dump to get<br>
those<br>
unfreed informations.<br>
<br>
I understand that for those 5 loss record, the 2nd, 3rd and 4th are true<br>
for all examples, but where do 1st and 5th ones come from? Also, the<br>
-malloc_dump information shows that there are<br>
"[0]Total space allocated 37780 bytes",<br>
but valgrind gives the information as<br>
"==26628== still reachable: 132,828 bytes in 323 blocks"<br>
<br>
Why there is a big difference?<br>
<br>
</blockquote>
<br>
<br>
1 is fine. It is from PMPI setup, which has some bytes not freed from<br>
setting up the MPI<br>
processes. The last one looks like an unfreed header for a DA, which is<br>
strange.<br>
<br>
Matt<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Thanks very much!<br>
<br>
Rebecca<br>
<br>
Here is the message from valgrind of running ex29:<br>
==26628== 32 bytes in 2 blocks are still reachable in loss record 1 of 5<br>
==26628== at 0x4022AB8: malloc (vg_replace_malloc.c:207)<br>
==26628== by 0x86F9A78: MPID_VCRT_Create (mpid_vc.c:62)<br>
==26628== by 0x86F743A: MPID_Init (mpid_init.c:116)<br>
==26628== by 0x86D040B: MPIR_Init_thread (initthread.c:288)<br>
==26628== by 0x86CFF2D: PMPI_Init (init.c:106)<br>
==26628== by 0x8613D69: PetscInitialize (pinit.c:503)<br>
==26628== by 0x804B796: main (ex29.c:139)<br>
==26628==<br>
==26628==<br>
==26628== 156 (36 direct, 120 indirect) bytes in 1 blocks are definitely<br>
lost in loss record 2 of 5<br>
==26628== at 0x4022AB8: malloc (vg_replace_malloc.c:207)<br>
==26628== by 0x429B3E2: (within /lib/tls/i686/cmov/<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a>)<br>
==26628== by 0x429BC2D: __nss_database_lookup (in /lib/tls/i686/cmov/<br>
<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a>)<br>
==26628== by 0x4732FDB: ???<br>
==26628== by 0x473413C: ???<br>
==26628== by 0x4247D15: getpwuid_r (in /lib/tls/i686/cmov/<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a><br>
)<br>
==26628== by 0x424765D: getpwuid (in /lib/tls/i686/cmov/<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a>)<br>
==26628== by 0x8623509: PetscGetUserName (fuser.c:68)<br>
==26628== by 0x85E0CF0: PetscErrorPrintfInitialize (errtrace.c:68)<br>
==26628== by 0x8613E23: PetscInitialize (pinit.c:518)<br>
==26628== by 0x804B796: main (ex29.c:139)<br>
==26628==<br>
==26628==<br>
==26628== 40 bytes in 5 blocks are indirectly lost in loss record 3 of 5<br>
==26628== at 0x4022AB8: malloc (vg_replace_malloc.c:207)<br>
==26628== by 0x429AFBB: __nss_lookup_function (in /lib/tls/i686/cmov/<br>
<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a>)<br>
==26628== by 0x4732FFB: ???<br>
==26628== by 0x473413C: ???<br>
==26628== by 0x4247D15: getpwuid_r (in /lib/tls/i686/cmov/<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a><br>
)<br>
==26628== by 0x424765D: getpwuid (in /lib/tls/i686/cmov/<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a>)<br>
==26628== by 0x8623509: PetscGetUserName (fuser.c:68)<br>
==26628== by 0x85E0CF0: PetscErrorPrintfInitialize (errtrace.c:68)<br>
==26628== by 0x8613E23: PetscInitialize (pinit.c:518)<br>
==26628== by 0x804B796: main (ex29.c:139)<br>
==26628==<br>
==26628==<br>
==26628== 80 bytes in 5 blocks are indirectly lost in loss record 4 of 5<br>
==26628== at 0x4022AB8: malloc (vg_replace_malloc.c:207)<br>
==26628== by 0x428839B: tsearch (in /lib/tls/i686/cmov/<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a>)<br>
==26628== by 0x429AF7D: __nss_lookup_function (in /lib/tls/i686/cmov/<br>
<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a>)<br>
==26628== by 0x4732FFB: ???<br>
==26628== by 0x473413C: ???<br>
==26628== by 0x4247D15: getpwuid_r (in /lib/tls/i686/cmov/<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a><br>
)<br>
==26628== by 0x424765D: getpwuid (in /lib/tls/i686/cmov/<a href="http://libc-2.7.so" target="_blank">libc-2.7.so</a>)<br>
==26628== by 0x8623509: PetscGetUserName (fuser.c:68)<br>
==26628== by 0x85E0CF0: PetscErrorPrintfInitialize (errtrace.c:68)<br>
==26628== by 0x8613E23: PetscInitialize (pinit.c:518)<br>
==26628== by 0x804B796: main (ex29.c:139)<br>
==26628==<br>
==26628==<br>
==26628== 132,796 bytes in 321 blocks are still reachable in loss record<br>
5<br>
of 5<br>
==26628== at 0x4022AB8: malloc (vg_replace_malloc.c:207)<br>
==26628== by 0x85EF3AC: PetscMallocAlign (mal.c:40)<br>
==26628== by 0x85F049B: PetscTrMallocDefault (mtr.c:194)<br>
==26628== by 0x81BCD3F: DACreate2d (da2.c:364)<br>
==26628== by 0x804BAFB: main (ex29.c:153)<br>
==26628==<br>
==26628== LEAK SUMMARY:<br>
==26628== definitely lost: 36 bytes in 1 blocks.<br>
==26628== indirectly lost: 120 bytes in 10 blocks.<br>
==26628== possibly lost: 0 bytes in 0 blocks.<br>
==26628== still reachable: 132,828 bytes in 323 blocks.<br>
==26628== suppressed: 0 bytes in 0 blocks.<br>
<br>
<br>
--<br>
(Rebecca) Xuefei YUAN<br>
Department of Applied Physics and Applied Mathematics<br>
Columbia University<br>
Tel:917-399-8032<br>
<a href="http://www.columbia.edu/%7Exy2102" target="_blank">www.columbia.edu/~xy2102</a> <<a href="http://www.columbia.edu/%7Exy2102" target="_blank">http://www.columbia.edu/%7Exy2102</a>> <<br>
<a href="http://www.columbia.edu/%7Exy2102" target="_blank">http://www.columbia.edu/%7Exy2102</a>><br>
<br>
<br>
<br>
</blockquote>
<br>
--<br>
What most experimenters take for granted before they begin their<br>
experiments<br>
is infinitely more interesting than any results to which their experiments<br>
lead.<br>
-- Norbert Wiener<br>
<br>
<br>
</blockquote>
<br>
<br>
--<br>
(Rebecca) Xuefei YUAN<br>
Department of Applied Physics and Applied Mathematics<br>
Columbia University<br>
Tel:917-399-8032<br>
<a href="http://www.columbia.edu/%7Exy2102" target="_blank">www.columbia.edu/~xy2102</a> <<a href="http://www.columbia.edu/%7Exy2102" target="_blank">http://www.columbia.edu/%7Exy2102</a>><br>
<br>
<br>
</blockquote>
<br>
<br>
--<br>
What most experimenters take for granted before they begin their experiments<br>
is infinitely more interesting than any results to which their experiments<br>
lead.<br>
-- Norbert Wiener<br>
<br>
</blockquote>
<br>
<br>
<br>
-- <br>
(Rebecca) Xuefei YUAN<br>
Department of Applied Physics and Applied Mathematics<br>
Columbia University<br>
Tel:917-399-8032<br>
<a href="http://www.columbia.edu/%7Exy2102" target="_blank">www.columbia.edu/~xy2102</a><br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>