itaps-parallel deletion of ghost entities

[Jason Kraftcheck]

Devine, Karen D wrote:

> You're right; the documentation isn't quite clear.  iMeshP_createGhostEnts
> should create ghosts right away -- it should do the communication when it is
> called.  But it also has to store the ghosting rules so that, if
> repartitioning or mesh modification of an existing mesh with ghosting
> occurs, the ghosts can be re-established after the repartitioning or mesh
> modification.  Re-establishing the ghosts in this second case is done by the
> end of iMeshP_syncMeshAll or iMeshP_syncPartitionAll; it can be done
> incrementally as, say, the mesh modification is done, but we guarantee only
> that ghosting is re-established by the end of the sync functions.

It isn't just that the documentation is unclear.  I think the API would be
better (simpler, easier to understand and use, etc.) if
iMeshP_createGhostEnts (and similarly iMeshP_deleteGhostEnts) only did one
of these two operations: either a) add a rule *or* b) communicate ghost
entities.

For a)
 o remove the implicit calls to iMeshP_syncMeshAll that every implementation
   will need to have at the end of iMeshP_createGhostEnts and
   iMeshP_deleteGhostEnts.
 o iMeshP_createGhostEnts and iMeshP_deleteGhostEnts do no communiction
 o Application must call iMeshP_syncMeshAll to communicate ghost entites.

For b)
 o remove iMeshP_syncMeshAll and iMeshP_ghostEntInfo and the concept of
   "ghosting rules".
 o iMeshP_createGhostEnts, iMeshP_deleteGhostEnts, etc. handle all
   communication.
 o after modifying mesh, application just calls iMeshP_createGhostEnts again
   to ghost any new entities, clean up remote references to deleted
   entities, etc.