[petsc-users] Providing Interpolation/Injections to Multigrid infrastructure

[Barry Smith]

> On Feb 12, 2016, at 9:20 AM, Paul T. Bauman <ptbauman at gmail.com> wrote:
> 
> Hi Barry,
> 
> On Thu, Feb 11, 2016 at 2:45 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Feb 11, 2016, at 1:36 PM, Boris Boutkov <borisbou at buffalo.edu> wrote:
> >
> > Hello All,
> >
> > I'm currently working on hooking into the PETSc multi-grid infrastructure by creating my own DMShell and providing my own interpolation and injection operators. The issue I am currently facing is that while PCSetUp_MG is running, PETSc begins by attempting to create an interpolation (through DMCreateInterpolation) for the finest grid that I am passing it.
> >
> > What would be a good way to try and let the PCSetUp_MG procedure know the first mesh it is receiving is the finest one, so there should be no interpolation provided to it?
> 
>   Boris,
> 
>   I do not understand the question, perhaps it is a matter of language.
> Say you have two levels total: call them coarse and fine. the PCSetUp_MG will try to create a single interpolation "for/to the fine grid" FROM the coarse grid. This is necessary to do the geometric multigrid. You would then through your DMSHELL provide the function that creates the interpolation matrix.  The PCSetUp_MG will not attempt to create an interpolation "for the coarse mesh" since there is no mesh coarser than the coarse mesh. So
> PCSetUp_MG() should be trying to create only the needed interpolations, so you should not need to tell them not to try to create certain ones.
> 
> The information I think we're missing is, more or less, the order of operations during the MG setup and how PETSc is expecting us to behave with our DMShell implementation. I'll stick with your coarse and fine grid. On our side, we can construct the coarse and the fine grids, supplying the DMs and accompanying PetscSections, at simulation setup time since we know what our grid hierarchy is (we're focusing on hierarchical meshes to start with). Then, for DMRefine and DMCoarsen, we can just return the correct DM that we've built. (Maybe this the wrong way to do it?) We also have implementations of DMCreateInterpolation and DMCreateInjection for moving between each of the levels.

   This is an ok way to do it.
> 
> Our main confusion is, once we've set up of all that, which of the DMs we've constructed do we go ahead and give to SNES? The fine or the coarse?

  You pass the finest to SNES because that is the one SNES uses for the nonlinear solve.  The linear multigrid solver will then call your DMCoarsen() to get the coarser DMs, so each time it is called you just pass the next coarser one from the DM that is the input argument to your DMCoarsen.

> Is it expected that we should be checking the DM supplied to, e.g., DMCreateInterpolation for which of the operators PETSc is wanting?

  Absolutely. It gets passed two DMS and based on those two DMS you need to pass back the correct interpolation.

> We were guessing that we should supply the fine DM to SNES (and then not expecting DMCreateInterpolation to be called yet).
> 
> Clearly, there's some setup steps we're missing or we're just going about this all the wrong way. Is there an existing example of setting up a problem using a DMShell? I poked a little but couldn't find one.

  Unfortunately not.

  You are over thinking this. It is actually not complicated or tricky. You might try running the two level one in the debugger (on one process) and put break points in all your DMSHELL call back functions, then you can check exactly what arguments they are being called with.

  Barry

> 
> Thanks much.
> 
> Best,
> 
> Paul