[petsc-dev] DMKSP and pcfieldsplit

[Lawrence Mitchell]

Dear petsc-dev,

95dbaa6faf01fdfd99114b7c9e5668e4b2aa754d

which does this:

commit 95dbaa6faf01fdfd99114b7c9e5668e4b2aa754d
Author: Matthew G. Knepley <knepley at gmail.com>
Date:   Fri Oct 26 14:12:14 2018 -0400

    PC: FieldSplit must copy DMKSP routines to subDM

diff --git a/src/ksp/pc/impls/fieldsplit/fieldsplit.c b/src/ksp/pc/impls/fieldsplit/fieldsplit.c
index b6a84ea342..f43f831204 100644
--- a/src/ksp/pc/impls/fieldsplit/fieldsplit.c
+++ b/src/ksp/pc/impls/fieldsplit/fieldsplit.c
@@ -386,6 +386,13 @@ static PetscErrorCode PCFieldSplitSetDefaults(PC pc)
           ierr = PetscObjectSetOptionsPrefix((PetscObject)(dms[i]), prefix);CHKERRQ(ierr);
           ierr = KSPSetDM(ilink->ksp, dms[i]);CHKERRQ(ierr);
           ierr = KSPSetDMActive(ilink->ksp, PETSC_FALSE);CHKERRQ(ierr);
+          {
+            PetscErrorCode (*func)(KSP,Mat,Mat,void*);
+            void            *ctx;
+
+            ierr = DMKSPGetComputeOperators(pc->dm, &func, &ctx);CHKERRQ(ierr);
+            ierr = DMKSPSetComputeOperators(dms[i],  func,  ctx);CHKERRQ(ierr);
+          }
           ierr = PetscObjectIncrementTabLevel((PetscObject)dms[i],(PetscObject)ilink->ksp,0);CHKERRQ
           ierr = DMDestroy(&dms[i]);CHKERRQ(ierr);
         }

Breaks many uses of PCMG and fieldsplit for us.

I would like to try and understand how best to fix it. My understanding of the way this stuff works is that when I say:

KSPSetComputeOperators

I provide a callback function (which I control), and some user-defined context (which I also control).

What this code is doing (which previously it did not), is to transfer this callback and context inside PCFieldsplit from a DM defining the full problem onto a DM from DMCreateFieldDecomposition (i.e. only part of the problem).

This seems like it is tremendously fragile, because PETSc has no way of knowing that the context for the full DM (which is just a void *) is valid for the subdm. In particular, in my case, I think it turns out not to be.

What is the right way to think about fixing this? I think that if the DM can create a field decomposition, it should also be able to transfer user callbacks from the parent to the subdms.

That way, the DM (IOW me) can control how that happens.

Or do I have the way this is supposed to work all wrong. Is the ComputeOperators callback I set meant to be completely agnostic, and it should get all the problem-specific information from the DM?

Cheers,

Lawrence