Hi Matthew,<div><br></div><div>Thanks,</div><div><br></div><div>I added the function PetscMallocDump() into the code after calling KSPSolve():</div><div><br></div><div><font color="#ff0000"> </font>(6) after calling KSPSolve()</div>
<div><br></div><div> ierr = PetscMallocGetCurrentUsage(&space);CHKERRQ(ierr);<br> ierr = PetscPrintf(comm,"Current space PetscMalloc()ed %G M\n", space/(1024*1024));CHKERRQ(ierr);<br> ierr = PetscMallocGetMaximumUsage(&space);CHKERRQ(ierr);<br>
ierr = PetscPrintf(comm,"Max space PetscMalloced() %G M\n", space/(1024*1024));CHKERRQ(ierr);<br> ierr = PetscMemoryGetCurrentUsage(&space);CHKERRQ(ierr);<br> ierr = PetscPrintf(comm,"Current process memory %G M\n", space/(1024*1024));CHKERRQ(ierr);<br>
ierr = PetscMemoryGetMaximumUsage(&space);CHKERRQ(ierr);<br> ierr = PetscPrintf(comm,"Max process memory %G M\n", space/(1024*1024));CHKERRQ(ierr);</div><div> <font color="#ff0000">ierr = PetscMallocDump(PETSC_NULL);CHKERRQ(ierr);</font></div>
<div><br></div><div> Current space PetscMalloc()ed 290.952 M</div> Max space PetscMalloced() 593.367 M<br> Current process memory 306.852 M<br> Max process memory 301.441 M<div><br></div><div>The printed detailed petscmalloc information is attached. The output seems too many lines to understand. How to understand this information?</div>
<div><br></div><div><br></div><div><br><div class="gmail_quote">On Tue, May 28, 2013 at 6:05 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="h5">On Tue, May 28, 2013 at 5:54 AM, Fande Kong <span dir="ltr"><<a href="mailto:Fande.Kong@colorado.edu" target="_blank">Fande.Kong@colorado.edu</a>></span> wrote:<br>
</div></div><div class="gmail_extra"><div class="gmail_quote"><div><div class="h5">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Smith,<br><br>Thank you very much. According to your suggestions and information, I added these functions into my code to measure the memory usage. Now I am confused, since the small problem needs large memory.<br>
<br>I added the function PetscMemorySetGetMaximumUsage() immediately after PetscInitialize(). And then I added the following code into several positions in the code (before & after setting up unstructured mesh, before & after KSPSetUp(), before & after KSPSolve(), and Destroy all stuffs):<br>
<br> PetscLogDouble space =0;<br> ierr = PetscMallocGetCurrentUsage(&space);CHKERRQ(ierr);<br> ierr = PetscPrintf(comm,"Current space PetscMalloc()ed %G M\n", space/(1024*1024));CHKERRQ(ierr);<br> ierr = PetscMallocGetMaximumUsage(&space);CHKERRQ(ierr);<br>
ierr = PetscPrintf(comm,"Max space PetscMalloced() %G M\n", space/(1024*1024));CHKERRQ(ierr);<br> ierr = PetscMemoryGetCurrentUsage(&space);CHKERRQ(ierr);<br> ierr = PetscPrintf(comm,"Current process memory %G M\n", space/(1024*1024));CHKERRQ(ierr);<br>
ierr = PetscMemoryGetMaximumUsage(&space);CHKERRQ(ierr);<br> ierr = PetscPrintf(comm,"Max process memory %G M\n", space/(1024*1024));CHKERRQ(ierr);<br><br><br>In order to measure the memory usage, I just used only one core (mpirun -n 1 ./program ) to solve a small problem with 12691 mesh nodes (the freedom is about 12691*3= 4 *10^4 ). I solve the linear elasticity problem by using FGMRES preconditioned by multigrid method (PCMG). I use all petsc standard routines except that I construct coarse matrix and interpolation matrix by myself. I used the following run script to set up solver and preconditioner:<br>
<br>mpirun -n 1 ./linearElasticity -ksp_type fgmres -pc_type mg -pc_mg_levels 2 -pc_mg_cycle_type v -pc_mg_type multiplicative -mg_levels_1_ksp_type richardson -mg_levels_1_ksp_max_it 1 -mg_levels_1_pc_type asm -mg_levels_1_sub_ksp_type preonly -mg_levels_1_sub_pc_type ilu -mg_levels_1_sub_pc_factor_levels 4 -mg_levels_1_sub_pc_factor_mat_ordering_type rcm -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 0.1 -mg_coarse_ksp_max_it 10 -mg_coarse_pc_type asm -mg_coarse_sub_ksp_type preonly -mg_coarse_sub_pc_type ilu -mg_coarse_sub_pc_factor_levels 2 -mg_coarse_sub_pc_factor_mat_ordering_type rcm -ksp_view -log_summary -pc_mg_log<br>
<br><br> I got the following results:<br><br>(1) before setting up mesh,<br><br>Current space PetscMalloc()ed 0.075882 M<br>Max space PetscMalloced() 0.119675 M<br>Current process memory 7.83203 M<br>Max process memory 0 M<br>
<br>(2) after setting up mesh,<br><br>Current space PetscMalloc()ed 16.8411 M<br>Max space PetscMalloced() 22.1353 M<br>Current process memory 28.4336 M<br>Max process memory 33.0547 M<br><br>(3) before calling KSPSetUp()<br>
<br>Current space PetscMalloc()ed 16.868 M<br>Max space PetscMalloced() 22.1353 M<br>Current process memory 28.6914 M<br>Max process memory 33.0547 M<br><br><br>(4) after calling KSPSetUp()<br><br>Current space PetscMalloc()ed 74.3354 M<br>
Max space PetscMalloced() 74.3355 M<br></blockquote><div><br></div></div></div><div>This makes sense. It is 20M for your mesh, 20M</div><div>for the Krylov space on the fine level, and I am guessing</div><div>35M for the Jacobian and the ILU factors.</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Current process memory 85.6953 M<br>Max process memory 84.9258 M<br>
<br>(5) before calling KSPSolve()<br><br>Current space PetscMalloc()ed 74.3354 M<br>Max space PetscMalloced() 74.3355 M<br>
Current process memory 85.8711 M<br>Max process memory 84.9258 M<br> <br>(6) after calling KSPSolve()<br></blockquote><div><br></div></div><div>The question is what was malloc'd here. There is no way we could</div><div>
tell without seeing the code and probably running it. I suggest</div><div>using <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscMallocDump.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscMallocDump.html</a></div>
<div>to see what was allocated. The solvers tend not to allocated during</div><div>the solve, as that is slow. So I would be inclined to check user code first.</div><div><br></div><div> Matt</div><div><div class="h5">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Current space PetscMalloc()ed 290.952 M<br>Max space PetscMalloced() 593.367 M<br>
Current process memory 306.852 M<br>
Max process memory 301.441 M<br><br>(7) After destroying all stuffs <br><br>Current space PetscMalloc()ed 0.331482 M<br>Max space PetscMalloced() 593.367 M<br>Current process memory 67.2539 M<br>Max process memory 309.137 M<br>
<br><br>So my question is why/if I need so much memory (306.852 M) for so small problem (freedom: 4*10^4). Or is it normal case? Or my run script used to set up solver is not reasonable?<br><br><br>Regards,<br><br>Fande Kong,<br>
<br>Department of Computer Science<br>University of Colorado Boulder<br><br><br><br><br><br><br><br><br><blockquote class="gmail_quote"></blockquote><br><br><div class="gmail_quote">On Mon, May 27, 2013 at 9:48 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
There are several ways to monitor the memory usage. You can divide them into two categories: those that monitor how much memory has been malloced specifically by PETSc and how much is used totally be the process.<br>
<br>
PetscMallocGetCurrentUsage() and PetscMallocGetMaximumUsage() which only work with the command line option -malloc provide how much PETSc has malloced.<br>
<br>
PetscMemoryGetCurrentUsage() and PetscMemoryGetMaximumUsage() (call PetscMemorySetGetMaximumUsage() immediately after PetscInitialize() for this one to work) provide total memory usage.<br>
<br>
These are called on each process so use a MPI_Reduce() to gather the total memory across all processes to process 0 to print it out. Suggest calling it after the mesh as been set up, then call again immediately before the XXXSolve() is called and then after the XXXSolve() is called.<br>
<br>
Please let us know if you have any difficulties.<br>
<br>
As always we recommend you upgrade to PETSc 3.4<br>
<br>
Barry<br>
<br>
<br>
<br>
On May 27, 2013, at 10:22 PM, Fande Kong <<a href="mailto:fande.kong@colorado.edu" target="_blank">fande.kong@colorado.edu</a>> wrote:<br>
<br>
> Hi all,<br>
><br>
> How to measure the memory usage of the application built on the Petsc? I am now solving linear elasticity equations with fgmres preconditioned by two-level method, that is, preconditioned by multigrid method where on each level the additive Schwarz method is adopted. More than 1000 cores are adopted to solve this problem on the supercomputer. When the total freedom of the problem is about 60M, the application correctly run and produce correct results. But when the total freedom increases to 600M, the application abort and say there is not enough memory ( the system administrator of the supercomputer told me that my application run out memory).<br>
><br>
> Thus, I want to monitor the memory usage dynamically when the application running. Are there any functions or strategies that could be used for this purpose?<br>
><br>
> The error information is attached.<br>
><br>
> Regards,<br>
<span><font color="#888888">> --<br>
> Fande Kong<br>
> Department of Computer Science<br>
> University of Colorado at Boulder<br>
> <solid3dcube2.o1603352><configure and make log.zip><br>
<br>
</font></span></blockquote></div><br>
</blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Fande Kong</div>Department of Computer Science<div>University of Colorado at Boulder</div>
</div>