[petsc-users] DMPlex ghost cell indexing

[Matthew Knepley]

On Sat, Jul 2, 2022 at 6:20 PM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Dear PETSc users,
>
> I have some general novice questions regarding how to properly access a
> distributed DMPLEX vector that has ghost cells for a finite volume use case.
>
> My process for setup
> 1) generated a simple box mesh using DMPlexCreateBoxMesh
> 2) distribute the mesh using DMPlexDistribute (at this point everything
> looks fine when I View (as paraview file) and I can see the partitions)
> 3) I generate a section with 5 variables located at the cells. (I don't
> think it matters if I use 5 fields or 1 field with 5 components?) I can
> alternatively do this using the PetscFV variable with 5 components.
> 4)  I then have the dm setup properly (with no ghost cells at this point)
> and I use DMCreateGlobalVector and DMCreateLocalVector to get the Vectors
> to work with
> 5) I'm just setting a quadratic IC. The overall loop over the cells is
> constructed via getting the cell stratum which has the Cell point list and
> then using the SectionOffset to assign each of the components at each cell
> using DMPlexComputeCellGeometryFVM to get the cell centroid.
>
> As far as I can tell this is an adequate solution but now I'm moving into
> learning how to access neighboring cells for flux calculations which isn't
> just the local cell but requires access to neighboring cells which at some
> point will be located on neighboring processors and synched to be
> accessible. When I did this for a DMDA it was pretty straightforward
> insofar as I just needed to call DMDAGetCorners and the LocalVector
> indexing would allow the stencil to extend gracefully into the ghost
> values.
>
> My general question is how do I achieve a similar loop structure for
> DMPlex as far as the indexing over the owned cells while being able to
> access the neighbor halo cells. I have been looking at example 52 and 11 in
> the ts tutorial examples but I'm struggling to extract exactly what I need.
>
> To boil it down into 3 starting questions
> 1) What is the preferred way to setup the cells and then access the
> internal cells for say N cell halos for the boundaries between processors
> in a DMPlex?
>

If you want a halo during distribution, you can give the "overlap" argument
to DMPlexDistribute().


> 2) What is the distinction between the overlap argument in
> DMPlexDistribute and DMPlexConstructGhostCells (which I see used in the
> examples)? (at least based on my tests it seems GhostCells is generating
> cells at the overall boundaries not the internal partition boundaries?)
>

The examples use both. DMPlexConstructGhostCells() puts a layer of cells
around the domain boundary that are special. They have no cell boundary,
just the face they are attached to. The overlap adds true cells to each
domain across the parallel boundary.


> 3) Once I do have a DMPlex object that has the appropriate halos, how
> would I get the access range for those cells? Based on my current
> assumption the Stratum range for the Cells would include the ghost cells as
> well which I would like to avoid?
>

I use
https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetSimplexOrBoxCells
to avoid the FV ghost cells. However, you can also do it by hand by
checking the type, or use
https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGhostCellStratum

  Thanks,

     Matt


> (Ultimately in the examples I see cases where there is the synch step for
> Local to Global and vice versa but I'm trying to just get to the point
> where I have the DM setup properly to handle the internal mesh boundaries
> and have the correct indexing from the StratumGet commands.)
>
> Any advice and corrections would be welcome.
>
> Sincerely
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220703/104f5d43/attachment-0001.html>