itaps-parallel deletion of ghost entities

[Devine, Karen D]

On 11/4/08 11:42 AM, "Jason Kraftcheck" <kraftche at cae.wisc.edu> wrote:

> Devine, Karen D wrote:
>
>> You're right; the documentation isn't quite clear.  iMeshP_createGhostEnts
>> should create ghosts right away -- it should do the communication when it is
>> called.  But it also has to store the ghosting rules so that, if
>> repartitioning or mesh modification of an existing mesh with ghosting
>> occurs, the ghosts can be re-established after the repartitioning or mesh
>> modification.  Re-establishing the ghosts in this second case is done by the
>> end of iMeshP_syncMeshAll or iMeshP_syncPartitionAll; it can be done
>> incrementally as, say, the mesh modification is done, but we guarantee only
>> that ghosting is re-established by the end of the sync functions.
>
> It isn't just that the documentation is unclear.  I think the API would be
> better (simpler, easier to understand and use, etc.) if
> iMeshP_createGhostEnts (and similarly iMeshP_deleteGhostEnts) only did one
> of these two operations: either a) add a rule *or* b) communicate ghost
> entities.
>
> For a)
>  o remove the implicit calls to iMeshP_syncMeshAll that every implementation
>    will need to have at the end of iMeshP_createGhostEnts and
>    iMeshP_deleteGhostEnts.
>  o iMeshP_createGhostEnts and iMeshP_deleteGhostEnts do no communiction
>  o Application must call iMeshP_syncMeshAll to communicate ghost entites.

IMeshP_syncMeshAll and iMeshP_syncPartitionAll are not needed unless mesh
modification and/or repartitioning is done.  So an application with a static
parallel mesh can create ghosts with iMeshP_createGhostEnts without ever
calling iMeshP_syncMeshAll.  This approach seems natural.  I'd hate to
require two calls for applications that are doing simple things.


> For b)
>  o remove iMeshP_syncMeshAll and iMeshP_ghostEntInfo and the concept of
>    "ghosting rules".
>  o iMeshP_createGhostEnts, iMeshP_deleteGhostEnts, etc. handle all
>    communication.
>  o after modifying mesh, application just calls iMeshP_createGhostEnts again
>    to ghost any new entities, clean up remote references to deleted
>    entities, etc.

IMeshP_syncMeshAll does more than re-establish ghosts (e.g., it waits for
all the asynchronous message traffic resulting from mesh modification to
clear).  So we can't eliminate it.

Also, the ghosting "rules" were meant to give a natural way for applications
to specify ghosts; they are the way most applications think about their
ghosts.  So I'm reluctant to remove that concept and force applications to
list exactly which entities must be communicated to other parts.

I can't speak well to the mesh modification case, but for partitioning, it
is nice to re-establish the ghosts for the application.  Say the application
has a mesh that has ghost entities.  We repartition that mesh and migrate it
to its new parts for the application.  It seems natural that we should also
call iMeshP_syncPartitionAll to re-establish the ghosts created by
iMeshP_createGhostEnts for the application.  In short, the partitioner
should take a valid, useful mesh and return an equivalent but redistributed
valid, useful mesh.


I don't understand what the confusion would be for the application.  The
application calls iMeshP_createGhostEnts with a distributed mesh and an
easy-to-specify ghosting rule, and gets back a distributed mesh with layers
of ghost entities.  The application later does, say, some repartitioning on
a valid mesh with layers of ghosts, and gets back a redistributed valid mesh
with the equivalent layers of ghosts.

Karen