[petsc-users] ILU, ASM, GMRES and memory

[francois pacull]

Thanks a lot Barry,
I did include the new function PCASMFreeSpace to asm.c and compile it... 
I will measure the effect on memory next week.
Regards,
francois.


Barry Smith wrote:
>   You will still need to delete your reference to the pmat by calling MatDestroy(pmat); after you have called KSPSetOperators().
>
>    Barry
>
> On Feb 18, 2011, at 9:42 AM, Barry Smith wrote:
>
>   
>> On Feb 18, 2011, at 9:22 AM, francois pacull wrote:
>>
>>     
>>> Dear PETSc team,
>>>
>>> I am using a gmres solver along with an additive Schwarz preconditioner and an ILU factorization within the sub-domains: ksp GMRES + pc ASM + subksp PREONLY + subpc ILU (MAT_SOLVER_PETSC). Also, I am using a preconditioner matrix Pmat that is different from the linear system operator matrix Amat.
>>>
>>> So, from my understanding, just after the end of the ILU factorization (for example, just after a call to PCSetUp(subpc) and before a call to KSPSolve(ksp,...)), the rank i process holds in the memory:
>>> 1 - the local rows of Amat (ksp&pc's linear system matrix)
>>> 2 - the local rows of Pmat (ksp&pc's precond matrix)
>>> 3 - the sub-domain preconditioner operator, P[i], which is the local diagonal block of Pmat augmented with the overlap (subksp&subpc's matrix, linear system matrix = precond matrix)
>>> 4 - the incomplete factorization of P[i] (subpc's ILU matrix)
>>>
>>> Is it correct?
>>> If it is, how can I destroy parts 2 and 3, Pmat and the P[i]'s, in order to save some memory space for the Arnoldi vectors?
>>>       
>>   You will need to "hack" slightly to get the affect. Edit src/ksp/pc/impls/asm/asm.c  and add a new function 
>>
>>   PCASMFreeSpace(PC pc)
>>   {
>>  PC_ASM         *osm = (PC_ASM*)pc->data;
>>  PetscErrorCode ierr;
>>
>>     if (osm->pmat) {
>>    if (osm->n_local_true > 0) {
>>      ierr = MatDestroyMatrices(osm->n_local_true,&osm->pmat);CHKERRQ(ierr);
>>    }
>>  osm->pmat = 0;
>>  }
>>  if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;}
>>  return 0;
>> }
>> run make in that directory.
>>
>>  Now call this routine in your program AFTER calling KSPSetUp() and KSPSetUpOnBlocks() or SNESSetUp() but before KSPSolve() or SNESSolve().
>>
>>  report any problems to petsc-maint at mcs.anl.gov
>>
>>
>>     
>>> When Pmat is destroyed with MatDestroy, its corresponding memory space seems to be actually freed only when the ksp is destroyed?
>>>       
>>   Yes, all PETSc objects are reference counted and the KSP object keeps a reference to Pmat (actually the PC underneath keeps the reference.)
>>
>>     
>>> From what I remember, PCFactorSetUseInPlace will destroy the P[i]'s, but under the constraints that there is no fill-in and the natural matrix ordering is used?
>>>       
>>   Under those conditions the space in P[i] is reused for the factor, thus saving the space of the "incomplete factorization of P"
>>
>>
>>     
>>> Thanks for your help,
>>> francois.
>>>
>>>
>>>       
>
>
>