[petsc-dev] DMSetUseNatural() and DMPlexConstructGhostCells()

[Adrian Croucher]

hi

I am testing out the new DMSetUseNatural(), DMPlexNaturalToGlobalBegin() 
etc. functions- along the lines of 
src/dm/impls/plex/examples/tests/ex15.c, but kinda the other way around.

What I want to do is set initial conditions in vector on a distributed 
DM representing a finite volume mesh. If I just do this using 
VecSetValues() they all end up in the wrong places on the mesh- in the 
global order rather than the natural order, with results dependent on 
number of processors.

So I'm trying:

- calling DMSetUseNatural(dm, PETSC_TRUE) before distributing the DM
- creating a separate 'natural' vector, and using VecSetValues() to set 
its values from the program input
- using DMPlexNaturalToGlobalBegin() / DMPlexNaturalToGlobalEnd() to 
fill my 'global' vector from the natural one
- viewing the final global vector with a VTK viewer to make sure the 
values are ending up in the right physical locations on the mesh.

It is all working ok until I also put in a DMPlexConstructGhostCells() 
(just after distributing) to fill the 'ghost' label on the DM. If I do 
that, when it gets to the DMPlexNaturalToGlobalBegin() call, it errors 
out with the following:

[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: DM global to natural SF was not created.
You must call DMPlexSetUseNaturalSF() before DMPlexDistribute().

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.6-512-g8d76ee9  GIT 
Date: 2015-07-19 12:15:50 -0400
[0]PETSC ERROR: supermodel on a linux-gnu-c-opt named des108 by acro018 
Mon Jul 20 16:32:22 2015
[0]PETSC ERROR: Configure options --download-netcdf --download-exodusii 
--with-hdf5-dir=/usr --download-triangle --download-ptscotch
--download-chaco --download-ctetgen
[0]PETSC ERROR: #1 DMPlexNaturalToGlobalBegin() line 203 in 
/home/acro018/software/PETSc/code/src/dm/impls/plex/plexnatural.c
--------------------------------------------------------------------------

So it thinks I haven't called DMSetUseNatural(), but I have. (There is 
also a typo in the error message- it's saying I should call 
'DMPlexSetUseNaturalSF()', but I think it means DMSetUseNatural()).

It looks a bit like DMPlexConstructGhostCells() might be messing with 
the DM's sfNatural property? or does this not work the way I think it 
should?

- Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 84611