[petsc-users] ILU, ASM, GMRES and memory

[Barry Smith]

On Feb 18, 2011, at 9:22 AM, francois pacull wrote:

> Dear PETSc team,
> 
> I am using a gmres solver along with an additive Schwarz preconditioner and an ILU factorization within the sub-domains: ksp GMRES + pc ASM + subksp PREONLY + subpc ILU (MAT_SOLVER_PETSC). Also, I am using a preconditioner matrix Pmat that is different from the linear system operator matrix Amat.
> 
> So, from my understanding, just after the end of the ILU factorization (for example, just after a call to PCSetUp(subpc) and before a call to KSPSolve(ksp,...)), the rank i process holds in the memory:
> 1 - the local rows of Amat (ksp&pc's linear system matrix)
> 2 - the local rows of Pmat (ksp&pc's precond matrix)
> 3 - the sub-domain preconditioner operator, P[i], which is the local diagonal block of Pmat augmented with the overlap (subksp&subpc's matrix, linear system matrix = precond matrix)
> 4 - the incomplete factorization of P[i] (subpc's ILU matrix)
> 
> Is it correct?
> If it is, how can I destroy parts 2 and 3, Pmat and the P[i]'s, in order to save some memory space for the Arnoldi vectors?

   You will need to "hack" slightly to get the affect. Edit src/ksp/pc/impls/asm/asm.c  and add a new function 

   PCASMFreeSpace(PC pc)
   {
  PC_ASM         *osm = (PC_ASM*)pc->data;
  PetscErrorCode ierr;

     if (osm->pmat) {
    if (osm->n_local_true > 0) {
      ierr = MatDestroyMatrices(osm->n_local_true,&osm->pmat);CHKERRQ(ierr);
    }
  osm->pmat = 0;
  }
  if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;}
  return 0;
 }
run make in that directory.

  Now call this routine in your program AFTER calling KSPSetUp() and KSPSetUpOnBlocks() or SNESSetUp() but before KSPSolve() or SNESSolve().

  report any problems to petsc-maint at mcs.anl.gov


> When Pmat is destroyed with MatDestroy, its corresponding memory space seems to be actually freed only when the ksp is destroyed?

   Yes, all PETSc objects are reference counted and the KSP object keeps a reference to Pmat (actually the PC underneath keeps the reference.)

> From what I remember, PCFactorSetUseInPlace will destroy the P[i]'s, but under the constraints that there is no fill-in and the natural matrix ordering is used?

   Under those conditions the space in P[i] is reused for the factor, thus saving the space of the "incomplete factorization of P"

    
> 
> Thanks for your help,
> francois.
> 
>