[petsc-dev] DMMG without libpetscsnes?

[Jed Brown]

On Thu, 7 Jan 2010 08:34:40 -0600, Barry Smith <bsmith at mcs.anl.gov> wrote:
>    I would not like to see them split up again. Rather I would like to  
> see the linear part of DMMG put into KSP/PC/DMMG somehow (not exactly  
> sure how) so it can be properly nested and composed just like other  
> preconditioners. I'm sure you are doing something slightly cumbersome  
> now to put your DMMG inside the Schur fieldsplit.

Yes, and I don't like it, but it's acceptable because this thing is an
application instead of a library (the required functionality is
sufficiently well-defined that it doesn't need to be nicely composable).

>     Here is how I see it: DMMG's job is to "fill up" the appropriate  
> fields in the KSP, PC, MG objects, then when solving those fields are  
> used. I would like (somehow) the KSP, PC, MG objects to have the  
> mechanism that you can provide a DM to them and they can then "fill  
> up" their fields. But it is important we get this right and  
> compossible so I have hesitated to ever try it. The starting point  
> might be having an (optional) KSPSetDM() that would propagate the DM  
> into the PC and subPCs and subsubPCs etc where they would be used to  
> fill the slots if provided.

So it seems to me that the real difficulty is in mesh sequencing (not a
big deal for linear problems), else PCMGSetDM seems like the natural
thing.

Looking at the DMMG code more, I see that DMMGSetKSP has

      ierr = KSPSetFromOptions(dmmg[i]->ksp);CHKERRQ(ierr);

      /*  if the multigrid is being run with Galerkin then these matrices do not need to be created except on the finest level
          we do not take advantage of this because it may be that Galerkin has not yet been selected for the KSP object 
          These are also used if grid sequencing is selected for the linear problem. We should probably turn off grid sequencing
          for the linear problem */

but it is used later in the same function (before user code has a chance
to get in with PCMGSetGalerkin)

  if (func) {
    if (galerkin) {
      ierr = (*func)(dmmg[nlevels-1],dmmg[nlevels-1]->J,dmmg[nlevels-1]->B);CHKERRQ(ierr);
    } else {
      for (i=0; i<nlevels; i++) {
        ierr = (*func)(dmmg[i],dmmg[i]->J,dmmg[i]->B);CHKERRQ(ierr);
      }
    }
  }

so if the user does PCMGSetGalerkin instead of -pc_mg_galerkin, the
series of matrices will be assembled twice as separate matrices (and gdb
tells me that references to both sets of matrices are retained for the
life of the DMMG, which could be a lot of memory).


If we were to go with PCMGSetDM, I think the construction of coarse
operators should to be postponed until PCSetUp (but the KSPs can be
created early so they can be played with).

Jed