itaps-parallel Proposal for communicating tag data

[Jason Kraftcheck]

I propose a slight modification to existing push/scatter functions.  This ad
change adds flexibility and consistency with the other functions I propose
later.  I think we should add a destination tag argument to each function.
It may be the same as the source tag, preserving the current functionality.
 I know this was discussed before, but I don't recall why it wasn't
included.  It seems like a trivial addition because the parts of the code
using each handle are separate (the sending code uses 'src_tag' and the
receiving code uses 'dst_tag').

void iMeshP_pushTags(iMesh_Instance instance,
          const iMeshP_PartitionHandle partition_handle,
                /*in*/ iBase_TagHandle src_tag,
		/*in*/ iBase_TagHandle dst_tag,
                /*in*/ int entity_type,
                /*in*/ int entity_topo,
                /*out*/ int *err);

void iMeshP_pushTagsEnt(iMesh_Instance instance,
                 const iMeshP_PartitionHandle partition_handle,
                   /*in*/ iBase_TagHandle src_tag,
		   /*in*/ iBase_TagHandle dst_tag,
                   /*in*/ const iBase_EntityHandle *entities,
                   /*in*/ int entities_size,
                   /*out*/ int *err);

void iMeshP_iPushTags(iMesh_Instance instance,
                 const iMeshP_PartitionHandle partition_handle,
                 /*in*/ iBase_TagHandle src_tag,
		 /*in*/ iBase_TagHandle dst_tag,
                 /*in*/ int entity_type,
                 /*in*/ int entity_topo,
                /*out*/ iMeshP_RequestHandle *req,
                /*out*/ int *err);

void iMeshP_iPushTagsEnt(iMesh_Instance instance,
                     const iMeshP_PartitionHandle partition_handle,
                    /*in*/ iBase_TagHandle src_tag,
		    /*in*/ iBase_TagHandle dst_tag,
                    /*in*/ const iBase_EntityHandle *entities,
                    /*in*/ int entities_size,
                   /*out*/ iMeshP_RequestHandle *req,
                   /*out*/ int *err);

Next I propose to sets of functions for a pull/gather type functionality.
These are the two alternatives I recall being discussed (gather and reduce).
 I propose that we support both.

The 'pull' functions are difficult to use if if copies of an element exist
on more than two processors (including the owning processor.)  However, if
an entity is shared by only two processors (e.g. face elements on the
interfaces of a volume mesh), it degenerates to the opposite of the
iMeshP_(i)PushTags(Ent) functions (send from copy to owner).  Further, they
are capable of providing a very generalized communication mechanism if that
is necessary.  They are more complicated to use in that case, but I think
that is unavoidable: it is a complex operation.

The 'reduce' functions provide a simpler, more efficient, and easier to use
mechanism for 'pulling' multiple tag values.  Like MPI_reduce, a reduction
operation is specified by the application to transform the multiple tag
values into a single tag value of the same type and size.  These operations
are somewhat redundant given that they could be implemented using the 'push'
operations above.  However, given that they are simpler to use (and often
more efficient to implement), and given that this type of reduction
operation is fairly common in real applications (e.g. they were worth
including in MPI) I think they is are worth including.


/**\brief Synchronously send the tag data from copies of an entity to the
 *        owner.
 *
 * If N processors have a copy of an entity, including the owning processor,
 * send N-1 tag values to the owning processor, one for each non-owned copy
 * of the entity.
 *
 * Given the set E of owned entities for which this processor will be
 * receiving tag values, let the set S be the result of
 * iMeshP_getNumCopies for each entity in E.  Let M = max(S)-1
 * (subtract one for the 'copy' on this processor).  The size of the
 * destination tag must be at least M times that of the source tag.
 *
 * If it is known that each entity is shared with at most one
 * processor (e.g. if mesh is partitioned based on region elements and no
 * ghosting of face elements is done, then all face elements are shared
 * with at most one other processor) then the size of the source and
 * destination tags may be the same.  In this case, it is permissible for
 * the source and destination to be the same tag.
 *
 *\param mesh       The mesh instance
 *\param partition  The owning partition
 *\param src_tag    The tag values to send
 *\param dst_tag    The tag into which to write received values
 *\param entity_type Only communicate tag data for the subset of entities of
 *                   this type.
 *\param entity_topo Only communicate tag data for the subset of entities of
 *                   this topology.
 */
void iMeshP_pullTags( iMesh_instance mesh,
               /*in*/ iMeshP_PartitionHandle partition,
               /*in*/ iBase_TagHandle src_tag,
               /*in*/ iBase_TagHandle dst_tag,
               /*in*/ int entity_type,
               /*in*/ int entity_topo,
              /*out*/ int *err);

/**\brief Same as iMeshP_pullTags, except an explicit entity list is
 *        specified, rather than an implicit list by type or topology
 */
void iMeshP_pullTagsEntArr( iMesh_instance mesh,
               /*in*/ iMeshP_PartitionHandle partition,
               /*in*/ iBase_TagHandle src_tag,
               /*in*/ iBase_TagHandle dst_tag,
               /*in*/ const iBase_EntityHandle *entities,
               /*in*/ int entities_size,
              /*out*/ int *err);

/**\brief Same as iMeshP_pullTags, except independent communcation
 */
void iMeshP_iPullTags( iMesh_instance mesh,
               /*in*/ iMeshP_PartitionHandle partition,
               /*in*/ iBase_TagHandle src_tag,
               /*in*/ iBase_TagHandle dst_tag,
               /*in*/ int entity_type,
               /*in*/ int entity_topo,
              /*out*/ iMeshP_RequestHandle *req,
              /*out*/ int *err);

/**\brief Same as iMeshP_pullTagsEntArr, except independent communcation
 */
void iMeshP_iPullTagsEntArr( iMesh_instance mesh,
               /*in*/ iMeshP_PartitionHandle partition,
               /*in*/ iBase_TagHandle src_tag,
               /*in*/ iBase_TagHandle dst_tag,
               /*in*/ const iBase_EntityHandle *entities,
               /*in*/ int entities_size,
              /*out*/ iMeshP_RequestHandle *req,
              /*out*/ int *err);


/**\brief Do MPI_reduce-like operation on tag values.
 *
 * This is a collective/synchronous operation.
 *
 * For a given entity, apply the specified operation to reduce
 * the set of tag values on all processors for which a copy of
 * the entity exists to a single tag value and store that
 * value in a destination tag (possibly the same as the source tag)
 * on the processor owning the entity.
 *
 *\param mesh       The mesh interface
 *\param partition  The owning partition
 *\param src_tag    Tag values to reduce
 *\param dst_tag    Location to store reduced tag values.  Must
 *                  be same type and size as 'src_tag'.  May be
 *                  'src_tag'.
 *\param entity_type Only communicate tag data for the subset of entities of
 *                   this type.
 *\param entity_topo Only communicate tag data for the subset of entities of
 *                   this topology.
 */
void iMeshP_reduceTags( iMesh_instance mesh,
               /*in*/ iMeshP_PartitionHandle partition,
               /*in*/ iBase_TagHandle src_tag,
               /*in*/ iBase_TagHandle dst_tag,
               /*in*/ MPI_Op operation,
               /*in*/ int entity_type,
               /*in*/ int entity_topo,
              /*out*/ int *err);

/**\brief Same as iMeshP_reduceTags, except an explicit entity list is
 *        specified, rather than an implicit list by type or topology
 */
void iMeshP_reduceTagsEntArr( iMesh_instance mesh,
               /*in*/ iMeshP_PartitionHandle partition,
               /*in*/ iBase_TagHandle src_tag,
               /*in*/ iBase_TagHandle dst_tag,
               /*in*/ MPI_Op operation,
               /*in*/ const iBase_EntityHandle *entities,
               /*in*/ int entities_size,
              /*out*/ int *err);