[petsc-users] ILU, ASM, GMRES and memory

[Barry Smith]

  You will still need to delete your reference to the pmat by calling MatDestroy(pmat); after you have called KSPSetOperators().

   Barry

On Feb 18, 2011, at 9:42 AM, Barry Smith wrote:

> 
> On Feb 18, 2011, at 9:22 AM, francois pacull wrote:
> 
>> Dear PETSc team,
>> 
>> I am using a gmres solver along with an additive Schwarz preconditioner and an ILU factorization within the sub-domains: ksp GMRES + pc ASM + subksp PREONLY + subpc ILU (MAT_SOLVER_PETSC). Also, I am using a preconditioner matrix Pmat that is different from the linear system operator matrix Amat.
>> 
>> So, from my understanding, just after the end of the ILU factorization (for example, just after a call to PCSetUp(subpc) and before a call to KSPSolve(ksp,...)), the rank i process holds in the memory:
>> 1 - the local rows of Amat (ksp&pc's linear system matrix)
>> 2 - the local rows of Pmat (ksp&pc's precond matrix)
>> 3 - the sub-domain preconditioner operator, P[i], which is the local diagonal block of Pmat augmented with the overlap (subksp&subpc's matrix, linear system matrix = precond matrix)
>> 4 - the incomplete factorization of P[i] (subpc's ILU matrix)
>> 
>> Is it correct?
>> If it is, how can I destroy parts 2 and 3, Pmat and the P[i]'s, in order to save some memory space for the Arnoldi vectors?
> 
>   You will need to "hack" slightly to get the affect. Edit src/ksp/pc/impls/asm/asm.c  and add a new function 
> 
>   PCASMFreeSpace(PC pc)
>   {
>  PC_ASM         *osm = (PC_ASM*)pc->data;
>  PetscErrorCode ierr;
> 
>     if (osm->pmat) {
>    if (osm->n_local_true > 0) {
>      ierr = MatDestroyMatrices(osm->n_local_true,&osm->pmat);CHKERRQ(ierr);
>    }
>  osm->pmat = 0;
>  }
>  if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;}
>  return 0;
> }
> run make in that directory.
> 
>  Now call this routine in your program AFTER calling KSPSetUp() and KSPSetUpOnBlocks() or SNESSetUp() but before KSPSolve() or SNESSolve().
> 
>  report any problems to petsc-maint at mcs.anl.gov
> 
> 
>> When Pmat is destroyed with MatDestroy, its corresponding memory space seems to be actually freed only when the ksp is destroyed?
> 
>   Yes, all PETSc objects are reference counted and the KSP object keeps a reference to Pmat (actually the PC underneath keeps the reference.)
> 
>> From what I remember, PCFactorSetUseInPlace will destroy the P[i]'s, but under the constraints that there is no fill-in and the natural matrix ordering is used?
> 
>   Under those conditions the space in P[i] is reused for the factor, thus saving the space of the "incomplete factorization of P"
> 
> 
>> 
>> Thanks for your help,
>> francois.
>> 
>> 
>