[petsc-users] Memory growth issue

[Sanjay Govindjee]

1) Correct: Placing a WaitAll before the MPI_Barrier solve the problem 
in our send-get routine for OPENMPI
2) Correct: The problem persists with KSPSolve
3) Correct: WaitAll did not fix the problem in our send-get nor in 
KSPSolve when using MPICH

Also correct.  Commenting out the call to KSPSolve results in zero 
memory growth on OPENMPI.

On 5/30/19 11:59 AM, Smith, Barry F. wrote:
>     Thanks for the update. So the current conclusions are that using the Waitall in your code
>
> 1) solves the memory issue with OpenMPI in your code
>
> 2) does not solve the memory issue with PETSc KSPSolve
>
> 3) MPICH has memory issues both for your code and PETSc KSPSolve (despite) the wait all fix?
>
> If you literately just comment out the call to KSPSolve() with OpenMPI is there no growth in memory usage?
>
>
> Both 2 and 3 are concerning, indicate possible memory leak bugs in MPICH and not freeing all MPI resources in KSPSolve()
>
> Junchao, can you please investigate 2 and 3 with, for example, a TS example that uses the linear solver (like with -ts_type beuler)? Thanks
>
>
>    Barry
>
>
>
>> On May 30, 2019, at 1:47 PM, Sanjay Govindjee <s_g at berkeley.edu> wrote:
>>
>> Lawrence,
>> Thanks for taking a look!  This is what I had been wondering about -- my knowledge of MPI is pretty minimal and
>> this origins of the routine were from a programmer we hired a decade+ back from NERSC.  I'll have to look into
>> VecScatter.  It will be great to dispense with our roll-your-own routines (we even have our own reduceALL scattered around the code).
>>
>> Interestingly, the MPI_WaitALL has solved the problem when using OpenMPI but it still persists with MPICH.  Graphs attached.
>> I'm going to run with openmpi for now (but I guess I really still need to figure out what is wrong with MPICH and WaitALL;
>> I'll try Barry's suggestion of --download-mpich-configure-arguments="--enable-error-messages=all --enable-g" later today and report back).
>>
>> Regarding MPI_Barrier, it was put in due a problem that some processes were finishing up sending and receiving and exiting the subroutine
>> before the receiving processes had completed (which resulted in data loss as the buffers are freed after the call to the routine). MPI_Barrier was the solution proposed
>> to us.  I don't think I can dispense with it, but will think about some more.
>>
>> I'm not so sure about using MPI_IRecv as it will require a bit of rewriting since right now I process the received
>> data sequentially after each blocking MPI_Recv -- clearly slower but easier to code.
>>
>> Thanks again for the help.
>>
>> -sanjay
>>
>> On 5/30/19 4:48 AM, Lawrence Mitchell wrote:
>>> Hi Sanjay,
>>>
>>>> On 30 May 2019, at 08:58, Sanjay Govindjee via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>
>>>> The problem seems to persist but with a different signature.  Graphs attached as before.
>>>>
>>>> Totals with MPICH (NB: single run)
>>>>
>>>> For the CG/Jacobi          data_exchange_total = 41,385,984; kspsolve_total = 38,289,408
>>>> For the GMRES/BJACOBI      data_exchange_total = 41,324,544; kspsolve_total = 41,324,544
>>>>
>>>> Just reading the MPI docs I am wondering if I need some sort of MPI_Wait/MPI_Waitall before my MPI_Barrier in the data exchange routine?
>>>> I would have thought that with the blocking receives and the MPI_Barrier that everything will have fully completed and cleaned up before
>>>> all processes exited the routine, but perhaps I am wrong on that.
>>> Skimming the fortran code you sent you do:
>>>
>>> for i in ...:
>>>     call MPI_Isend(..., req, ierr)
>>>
>>> for i in ...:
>>>     call MPI_Recv(..., ierr)
>>>
>>> But you never call MPI_Wait on the request you got back from the Isend. So the MPI library will never free the data structures it created.
>>>
>>> The usual pattern for these non-blocking communications is to allocate an array for the requests of length nsend+nrecv and then do:
>>>
>>> for i in nsend:
>>>     call MPI_Isend(..., req[i], ierr)
>>> for j in nrecv:
>>>     call MPI_Irecv(..., req[nsend+j], ierr)
>>>
>>> call MPI_Waitall(req, ..., ierr)
>>>
>>> I note also there's no need for the Barrier at the end of the routine, this kind of communication does neighbourwise synchronisation, no need to add (unnecessary) global synchronisation too.
>>>
>>> As an aside, is there a reason you don't use PETSc's VecScatter to manage this global to local exchange?
>>>
>>> Cheers,
>>>
>>> Lawrence
>> <cg_mpichwall.png><cg_wall.png><gmres_mpichwall.png><gmres_wall.png>