[petsc-users] Question about GAMG and memory use

[Randall Mackie]

Hi Barry,

I've followed your suggestion to use PetscMallocDump after the call to KSPSolve and I've looked at the output.

What I find is that for a smaller sized model that I can run on my laptop, I don't see any increase in the memory after it is allocated the first pass through.

However, with this other much larger model, I do clearly see an increase in memory after each call to that particular KSPSolve:

[0]Total space allocated 1885177504 bytes
[0]Total space allocated 2034781200 bytes
[0]Total space allocated 2184383136 bytes
[0]Total space allocated 2333985072 bytes
[0]Total space allocated 2483587008 bytes


That model is too large to run on just 1 process, but I've got it down to 8 processes, and each process shows a similar increase in memory.

Just to confirm that it was happening in that KSPSolve call, I put an additional PetscMallocDump before and after, and I can confirm that it is clearly increasing the memory after the call to KSPSolve.

I've tried sorting through the output of that malloc dump, but the file is 4 Mbytes in size for process 0, so I've tried grepping for various things, like GAMG, etc, and I don't yet see what is causing the increase in memory.

Do you guys have any sophisticated scripts to parse the output with?

If you'd like the output file, I can put it somewhere, I just don't want to drop a 4 Mbyte file in everyone's inbox.

As another datapoint, I did save the matrix and vector to a binary file using Vec/Mat View, then wrote a little test program that read them in (Vec/Mat Load) and did the KSPSolve. I did not see any memory issues with that.

So, I am unsure what to do at this point, or how to proceed to track this down.

Any suggestions appreciated.


Randy


> On Mar 5, 2015, at 2:41 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Randy,
> 
>   I've not been able to reproduce this; let us know if get to the point of having something we can run and debug.
> 
>  Barry
> 
>> On Mar 4, 2015, at 7:45 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>> 
>> In my application, I am repeatedly calling KSPSolve with the following options:
>> 
>> -ksp_type gmres \
>> -pc_type gamg \
>> -pc_gamg_type agg \
>> -pc_gamg_agg_nsmooths 1\
>> 
>> 
>> each call is after the matrix and right hand side have been updated.
>> 
>> This works well in the sense that it solves the system in a reasonable number of steps, however, I have noticed that the memory footprint of the application increases by about 500 Mbytes after each call to KSPSolve (this is a big problem), and after several calls, I've maxed out the memory.
>> 
>> Is this expected behavior?
>> 
>> I've combed through my code looking to make sure I don't have any memory leaks, and so far I haven't found any (doesn't mean there are not there).
>> 
>> However, when I use another PC, like jacobi, just to compare, I don't see this memory issue, or if I comment out that call to KSPSolve (there is a lot of other stuff going on in the code besides this call), I don't see this issue.
>> 
>> I've tried to destroy the KSP after each solve and recreate it each time, but there still seems to be some memory getting added.
>> 
>> I've tried to distill this down to a smaller problem and test program, but so far I have been unsuccessful.
>> 
>> 
>> Is there a way to completely release the memory associated with the GAMG preconditioner after a call to KSPSolve?
>> 
>> 
>> Any other suggestions for tracking this down? I've run out of ideas.
>> 
>> 
>> Thanks in advance,
>> 
>> Randy
>> 
>> 
>> 
>> 
>