[petsc-dev] Trying to get DMShell usable for PFLOTRAN

[Richard Tran Mills]

Hi Jed (cc: petsc-dev, in case anyone else cares),

As we discussed on gchat, if you will grab

   https://bitbucket.org/rmills/petsc-dev
   https://bitbucket.org/rmills/pflotran-dev

you can see what I have been trying to do to get DMShell usable for 
wrapping our PFLOTRAN unstructured grids in a DMs.  All of the relevant 
PFLOTRAN changes are in discretization.F90.  As a first step, I'm only 
trying to get the global-to-local operations working using DMs.  I'm 
seeing the errors when I try to run a very simple unstructured grid 
problem in 'example_problems/umesh/usg_5x4x3'.  If I just run PFLOTRAN 
in there (one process, with no command-line arguments) it crashes with 
lots of complaints along the lines of

> [0]PETSC ERROR: PetscObjectGetFortranCallback() line 254 in 
> /Users/rmills/proj/petsc-dev_rmills/src/sys/objects/inherit.c
> [0]PETSC ERROR: ourglobaltolocalend() line 80 in 
> /Users/rmills/proj/petsc-dev_rmills/src/dm/impls/shell/ftn-custom/zdmshellf.c
> [0]PETSC ERROR: DMGlobalToLocalEnd() line 1668 in 
> /Users/rmills/proj/petsc-dev_rmills/src/dm/interface/dm.c

So it appears that I am doing something wrong.  I will admit that so far 
I've just blindly emulated the examples you already had in zdmshellf.c 
and I haven't taken the time to digest how the new Fortran callback 
mechanism works, so maybe I'm just doing something dumb with that.

On a somewhat related note, I've added a means to set a GlobalToLocal 
scatter context for the DMShell, which I do in the PFLOTRAN code, and 
the DMShellDefaultGlobalToLocalBegin/End, which just performs the 
scatter using that 'gtol' context that is stashed.  I am setting the 
'gtol' scatter and then specifying that the 
DMShellDefaultGlobalToLocalBegin/End be used.  I suppose it makes more 
sense for the user to not have to set this if the 'gtol' has been 
given.  Should I make the dm->ops->globaltolocalbegin point to 
DMShellDefaultGlobalToLocalBegin when the DMShell is created, and have 
that function check to see if the 'gtol' pointer is indeed set to something?

--Richard

-- 
Richard Tran Mills, Ph.D.
Computational Earth Scientist      | Joint Assistant Professor
Hydrogeochemical Dynamics Team     | EECS and Earth & Planetary Sciences
Oak Ridge National Laboratory      | University of Tennessee, Knoxville
E-mail: rmills at ornl.gov  V: 865-241-3198 http://climate.ornl.gov/~rmills