[petsc-users] Question about GAMG and memory use

[Barry Smith]

  Randy,

  Just email me the PetscMallocDump output file directly.

  Barry

> On Mar 6, 2015, at 12:10 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> 
> Hi Barry,
> 
> I've followed your suggestion to use PetscMallocDump after the call to KSPSolve and I've looked at the output.
> 
> What I find is that for a smaller sized model that I can run on my laptop, I don't see any increase in the memory after it is allocated the first pass through.
> 
> However, with this other much larger model, I do clearly see an increase in memory after each call to that particular KSPSolve:
> 
> [0]Total space allocated 1885177504 bytes
> [0]Total space allocated 2034781200 bytes
> [0]Total space allocated 2184383136 bytes
> [0]Total space allocated 2333985072 bytes
> [0]Total space allocated 2483587008 bytes
> 
> 
> That model is too large to run on just 1 process, but I've got it down to 8 processes, and each process shows a similar increase in memory.
> 
> Just to confirm that it was happening in that KSPSolve call, I put an additional PetscMallocDump before and after, and I can confirm that it is clearly increasing the memory after the call to KSPSolve.
> 
> I've tried sorting through the output of that malloc dump, but the file is 4 Mbytes in size for process 0, so I've tried grepping for various things, like GAMG, etc, and I don't yet see what is causing the increase in memory.
> 
> Do you guys have any sophisticated scripts to parse the output with?
> 
> If you'd like the output file, I can put it somewhere, I just don't want to drop a 4 Mbyte file in everyone's inbox.
> 
> As another datapoint, I did save the matrix and vector to a binary file using Vec/Mat View, then wrote a little test program that read them in (Vec/Mat Load) and did the KSPSolve. I did not see any memory issues with that.
> 
> So, I am unsure what to do at this point, or how to proceed to track this down.
> 
> Any suggestions appreciated.
> 
> 
> Randy
> 
> 
>> On Mar 5, 2015, at 2:41 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>> Randy,
>> 
>>  I've not been able to reproduce this; let us know if get to the point of having something we can run and debug.
>> 
>> Barry
>> 
>>> On Mar 4, 2015, at 7:45 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>>> 
>>> In my application, I am repeatedly calling KSPSolve with the following options:
>>> 
>>> -ksp_type gmres \
>>> -pc_type gamg \
>>> -pc_gamg_type agg \
>>> -pc_gamg_agg_nsmooths 1\
>>> 
>>> 
>>> each call is after the matrix and right hand side have been updated.
>>> 
>>> This works well in the sense that it solves the system in a reasonable number of steps, however, I have noticed that the memory footprint of the application increases by about 500 Mbytes after each call to KSPSolve (this is a big problem), and after several calls, I've maxed out the memory.
>>> 
>>> Is this expected behavior?
>>> 
>>> I've combed through my code looking to make sure I don't have any memory leaks, and so far I haven't found any (doesn't mean there are not there).
>>> 
>>> However, when I use another PC, like jacobi, just to compare, I don't see this memory issue, or if I comment out that call to KSPSolve (there is a lot of other stuff going on in the code besides this call), I don't see this issue.
>>> 
>>> I've tried to destroy the KSP after each solve and recreate it each time, but there still seems to be some memory getting added.
>>> 
>>> I've tried to distill this down to a smaller problem and test program, but so far I have been unsuccessful.
>>> 
>>> 
>>> Is there a way to completely release the memory associated with the GAMG preconditioner after a call to KSPSolve?
>>> 
>>> 
>>> Any other suggestions for tracking this down? I've run out of ideas.
>>> 
>>> 
>>> Thanks in advance,
>>> 
>>> Randy
>>> 
>>> 
>>> 
>>> 
>> 
>