itaps-parallel Ghost exchange prototype

[Tim Tautges]

Hi all,
   I propose the following function for exchanging ghost entities in a 
parallel mesh.  This is a concurrent call on all processors, and assumes 
the mesh already knows what's shared with other processors:

iMeshP_exchange_ghost_cells(iMesh_Instance instance,
           /*in*/ prefix_PartitionHandle partition_handle,
           int ghost_dim, int bridge_dim,
           int num_layers,
           int *err);

instance: mesh instance
partition_handle: partition handle (implicitly defines communicator)
ghost_dim: dimension of the entities to be exchanged
bridge_dim: intermediate entities, or the entities through which the 2nd
             order adjacencies are found; if -1, all dimensions =
             0..ghost_dim-1 are used to find 2nd order-adjacent entities
num_layers: number of layers of ghost entities to exchange; if 0,
             non-vertex entities on the interface are exchanged
             (in my implementation, I reuse ghost exchange code for
             finding out what the remote handles are for non-vertex ents
             on shared interfaces)
err: error flag

This is a fairly high-level definition, e.g. it doesn't specify which 
processors to exchange ghosts with.  I did it that way in order to allow 
a high degree of asychronicity (which is used extensively in my ghost 
exchange code).

My implementation has two additional arguments to this function:

store_remote_handles: indicating whether the local instance should store 
remote handles; I'm considering taking this out, since it doesn't make 
much sense to not store them

wait_all: if false, returns before receiving all posted receives. 
Again, I'm considering taking this out, since I currently haven't 
implemented a mechanism to come back and check for pending receives. 
I'm still thinking about that though.

Also, I noticed in the current draft interface there isn't a function to 
get local ghost entities.  That statement assumes that in our definition 
a ghost entity isn't a part boundary entity.  Should there be such a 
function?

Other comments?

- tim

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706