/*
------------------------------------------------
Background and Terminology:

Terminology -- Abstract Data Model:
-  The term "mesh" refers to an abstraction in the data model; 
   it does not imply a serial or parallel distribution.
-  The term "partition" refers to an assignment of a set of entities to 
   subsets; like a "mesh," it does not imply a serial or parallel 
   implementation.
-  An application may use one or more meshes.  
-  Parititions can create subsets of entities from one or more meshes.
-  Meshes can be subdivided by one or more partitions.
-  Partitions contain parts.  Parts contain the subsets of entities in the
   partition.

Terminology -- Parallelism
-  A "process" can be thought of as an MPI process. The
   number of processes can be considered to be the result of MPI_Comm_size.
   The rank of a process can be thought of as the result of MPI_Comm_rank.
   We will think in terms of processes rather than processors.  Initial
   implementations of the parallel interface will likely use MPI terminology
   directly; future implementations may accommodate other communication 
   paradigms and libraries.
-  Partitions have communicators associated with them.  These communicators
   can be thought of as MPI communicators.  
-  "Global" operations are operations performed with respect to a 
   partition's communicator.
-  "Local" operations are operations performed with respect to a part.
-  Part A "neighbors" Part B if A and B share entities of a given dimension;
   functions computing part neighbors will accept an entity type (possibly
   iBase_ALL_TYPES).

Terminology -- Interfaces
-  Each process has one or more "mesh instances."  A mesh instance can be
   thought of as a mesh database.  An implementation should support the 
   existence of more than one mesh instance per process (e.g., it should 
   always associate mesh data with a mesh instance).  However, we expect 
   applications would most often use only one mesh instance per process.
-  There is one root set per mesh instance.
-  Each process may have one or more partition handles.
-  A partition partitions entities in one mesh instance.  
   Partition instances know which mesh instance's entities they partition. 
   KDD:  I DOUBT WE NEED THIS KNOWLEDGE ANYMORE.
   KDD:  PERHAPS WE SHOULD NOT MAKE IT A REQUIREMENT BUT, RATHER, AN 
   KDD:  IMPLEMENTATION OPTION.
-  Entities in a mesh instance can be partitioned by one or more partitions.  
   Mesh instances know which partitions they are in.
-  Parts can be accessed by either part IDs (integers) or part handles.  

Assumptions that aren't currently causing heartburn:
-  Each part is wholly contained within a process.  
-  A process may have multiple parts.
-  Many iMesh functions that accept a mesh instance and an EntitySet handle 
   are also useful in the context of partition handles and part handles.  
   These functions will be reinterpreted so that they can accept either a 
   mesh instance and EntitySet handle, or a partition handle and a 
   part handle.  See Carl's Oct 15 email for more detail.
-  Entities in partitions can have unique global IDs.
   Uniqueness of global IDs is managed at the partition level.
-  For **each item** of functionality in this document, there will be a 
   function (or set of functions) to provide the functionality.  The 
   implementation can decide whether these functions should be simple 
   wrappers over existing data model functionality or more intrusive functions 
   in the implementation.
*/

/*------------------------------------------------------------------------*/
/*------------------------------------------------------------------------*/
/*                          Partition Functionality                       */
/*------------------------------------------------------------------------*/
/*------------------------------------------------------------------------*/

////////////////////////////////
//  Create a partition handle:  Given a mesh instance to be 
//  included in the partition and a communicator, return a partition
//  handle.  May have different creation routines for different 
//  communication systems; once created, the application shouldn't have to
//  worry about the communication system again.

    void prefix_createPartitionMPI(iMesh_Instance instance,
                    /*in*/  MPI_Comm communicator,
                    /*out*/ prefix_PartitionHandle *partition_handle,
                            int *err);
 
////////////////////////////////
//  Destroy a partition handle: Given a partition handle, destroy it and set
//  it to NULL.

    void prefix_destroyPartition(iMesh_Instance instance,
                    /*inout*/ prefix_PartitionHandle *partition_handle,
                              int *err);

////////////////////////////////
//  Given a partition handle, return its communicator.
    void prefix_getPartitionCommMPI(iMesh_Instance instance,
                    /*in*/  prefix_PartitionHandle partition_handle,
                    /*out*/ MPI_Comm *communicator,
                            int *err);
    
////////////////////////////////
//  Given a partition handle, return the mesh instance that is
//  included in the partition.  
//  KDD:  DO WE NEED THIS CAPABILITY?  IT IS HARD TO SPECIFY
//  KDD:  IF iMesh_Instance MUST ALWAYS BE FIRST ARG OF iMesh FUNCTIONS.  :)

////////////////////////////////
//  Given a mesh instance, return number of partition handles and 
//  all partition handles associated with the mesh instance.

    void prefix_getNumPartitions(iMesh_Instance instance,
                    /*out*/ int *num_partitions,
                            int *err);

    void prefix_getPartitions(iMesh_Instance instance,
             /*inout*/ prefix_PartitionHandle **partition_handle,
             /*inout*/ int *partition_handle_allocated, 
             /*out*/   int *partition_handle_size, 
                       int *err); 

////////////////////////////////
// Given a partition handle, return the total global number of parts 
// in the partition.

    void prefix_getNumPartsPar(iMesh_Instance instance,
                          const prefix_PartitionHandle partition_handle,
                          int *num_global_part, 
                          int *err); 

////////////////////////////////
//  Map from parts to processes:  
//  Given a partition handle and a part ID, return the rank of the 
//  process that owns the part.

    void prefix_getRank(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const int part_id,
             /*out*/   int *rank,
                       int *err); 
    void prefix_getRankArr(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const int *part_id,
             /*in*/    const int part_id_size,
             /*inout*/ int **rank, 
             /*inout*/ int *rank_allocated, 
                       int *err); 

////////////////////////////////
//  Map from processes to parts:  

//  +  Given a partition handle and an process rank, 
//     return the number of parts owned by the process.

    void prefix_getNumPartIds(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const int rank,
             /*out*/   int *num_pids,
                       int *err); 

    void prefix_getNumPartIdsArr(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const int *rank,
             /*in*/    const int rank_size,
             /*inout*/ int **num_pids, 
             /*inout*/ int *num_pids_allocated, 
                       int *err); 

//  +  Given a partition handle and a process rank,
//     return the part IDs owned by the process.

    void prefix_getPartIdsFromRank(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const int rank,
             /*inout*/ int **pids, 
             /*inout*/ int *pids_allocated, 
             /*out*/   int *pids_size, 
                       int *err); 

    void prefix_getPartIdsFromRankArr(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const int *rank,
             /*in*/    const int rank_size,
             /*inout*/ int **pids, 
             /*inout*/ int *pids_allocated, 
             /*out*/   int *pids_size, 
                       int *err); 

////////////////////////////////
//  Map from the process rank to part handles.
//  + Given a partition handle, return the number of on-process parts in the
//    partition.

    void prefix_getNumParts(iMesh_Instance instance,
                          const prefix_PartitionHandle partition_handle,
                          int *num_part, 
                          int *err); 

//  + Given a partition handle, return all on-process part handles in the
//    partition.

    void prefix_getParts(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
                       prefix_PartHandle *part_handles, 
                       int *err); 

////////////////////////////////
//  Provide global mesh information about a partition.  
//  Note that these functions may require communication and, thus, 
//  would have to be called by all processes in the partition handle.
//  Given a mesh instance and partition handle, return:
//  + Total number of entities with given type or topology in the partition.

    void prefix_getNumOfTypePar(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
                       const iBase_EntitySetHandle entity_set_handle,
                       const int entity_type, int *num_type, int *err);

    void prefix_getNumOfTopoPar(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
                       const iBase_EntitySetHandle entity_set_handle,
                       const int entity_topology, int *num_topo, int *err);

//  + Total number of entity sets in the partition.

    void prefix_getNumEntSetsPar(iMesh_Instance instance, 
                       const prefix_PartitionHandle partition_handle,
                       const iBase_EntitySetHandle entity_set_handle, 
                       const int num_hops, int *num_sets, int *err); 

//  KDD:  THIS FUNCTION WAS NOT ONE OF CARL'S SIXTEEN, BUT IT IS ANALOGOUS TO
//  KDD:  HIS FUNCTIONS.  DO WE NEED IT?

////////////////////////////////
//  Manage global IDs of entities that are partitioned.
//  + set/retrieve global IDs; generate global IDs; 
//    set/retrieve global ID size; map between entities and global IDs;
//    compare global IDs.  Since there was little disagreement with 
//    Vitus'/Karen's document on global IDs, we will use it in the first
//    draft with modifications to associate the "global" nature of the 
//    IDs with partitions; these changes address some of Tim's concerns.  
//    We will keep the helper functions defined, rather than require the 
//    application to manage them with special tags.
//  KDD:  The more I think about this, the more I think it should be a service,
//  KDD:  not part of the interface.  Perhaps it should be part of the dynamic
//  KDD:  load balancing service, since we are the only ones who care to use it 
//  KDD:  currently.
//  KDD:  Does anyone else agree?

/*------------------------------------------------------------------------*/
/*------------------------------------------------------------------------*/
/*                        Part Functionality                              */
/*------------------------------------------------------------------------*/
/*------------------------------------------------------------------------*/

////////////////////////////////
//  Given a partition handle, create a part and add it to the
//  partition on the process invoking the creation.  Return part handle.

    void prefix_createPart(iMesh_Instance instance,
                   /*in*/  prefix_PartitionHandle partition_handle,
                   /*out*/ prefix_PartHandle *part_handle,
                           int *err);
 
////////////////////////////////
// Given a partition handle and a part handle, remove the part
//  from the partition, destroy the part, and set the part handle to NULL.  

    void prefix_destroyPart(iMesh_Instance instance,
                    /*in*/ prefix_PartitionHandle partition_handle,
                 /*inout*/ prefix_PartHandle *part_handle,
                           int *err);
 
////////////////////////////////
//  Map between part handles and part IDs.
//  + Given a partition handle and a part handle, return the part ID.

    void prefix_getPartIdsFromPartHandle(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const prefix_PartHandle part_handle,
             /*out*/   int *part_id,
                       int *err); 

    void prefix_getPartIdsFromPartHandlesArr(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const prefix_PartHandle *part_handles,
             /*in*/    const int part_handles_size,
             /*inout*/ int **part_ids,
             /*inout*/ int *part_ids_allocated, 
                       int *err); 

//  + Given a partition handle and a part ID, return the part handle 
//    if on process; return error code otherwise.

    void prefix_getPartHandlesFromPartId(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    int part_id,
             /*out*/   prefix_PartHandle *part_handle,
                       int *err); 

    void prefix_getPartHandlesFromPartIdsArr(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const int *part_ids,
             /*in*/    const int part_ids_size,
             /*inout*/ prefix_PartHandle **part_handles,
             /*inout*/ int *part_handles_allocated, 
                       int *err); 

////////////////////////////////
//  Identify parts that neighbor a given part.
//  + Given a partition handle, a part handle, and an entity type, 
//    return the number of parts in the partition that neighbor the given part
//    (using the definition of "neighbor" above).

    void prefix_getNumPartNbors(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const prefix_PartHandle part_handle,
             /*in*/    int entity_type,
             /*out*/   int *num_part_nbors,
                       int *err); 

    void prefix_getNumPartNborsArr(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const prefix_PartHandle *part_handles,
             /*in*/    const int part_handles_size,
             /*in*/    int entity_type,
             /*inout*/ int **num_part_nbors,
             /*inout*/ int *num_part_nbors_allocated,
                       int *err); 

//  + Given a partition handle and a part handle, return the part IDs of all 
//    parts in the partition that neighbor the given part.  

    void prefix_getPartNbors(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const prefix_PartHandle part_handle,
             /*in*/    int entity_type,
             /*inout*/ int **part_nbor_ids,
             /*inout*/ int *part_nbor_ids_allocated,
             /*out*/   int *part_nbor_ids_size,
                       int *err); 

    void prefix_getPartNborsArr(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
             /*in*/    const prefix_PartHandle *part_handles,
             /*in*/    const int part_handles_size,
             /*in*/    int entity_type,
             /*inout*/ int **part_nbor_ids,
             /*inout*/ int *part_nbor_ids_allocated,
             /*out*/   int *part_nbor_ids_size,
                       int *err); 

////////////////////////////////
//  Provide part boundary info:
//  Note: Allow optional specification of desired entity types and 
//  topologies; allow neighboring part ID = -1 to count/include all qualifying 
//  boundary entities of part.
//  + Given a partition handle, a part handle, entity type and topology, and a
//    neighboring part ID, return the number of qualifying entities on 
//    the part boundary shared with the neighboring part ID.  

    void prefix_getNumPartBdryEntities(iMesh_Instance instance,
                      /*in*/  const prefix_PartitionHandle partition_handle, 
                      /*in*/  const prefix_PartHandle part_handle, 
                      /*in*/  const int requested_entity_type, 
                      /*in*/  const int requested_entity_topology, 
                      /*in*/  const int requested_nbor_part_id, 
                      /*out*/ int *num_nbor_entities, 
                              int *err); 

//  + Given a partition handle, a part handle, entity type and topology, and a 
//    neighboring part ID, return an array of entity handles for all qualifying
//    entities along the part boundary shared with the neighboring part ID.

    void prefix_getPartBdryEntities(iMesh_Instance instance,
                      /*in*/  const prefix_PartitionHandle partition_handle, 
                      /*in*/  const prefix_PartHandle part_handle, 
                      /*in*/  const int requested_entity_type, 
                      /*in*/  const int requested_entity_topology, 
                      /*in*/  const int requested_nbor_part_id, 
                   /*inout*/  iBase_EntityHandle **entity_handles,
                   /*inout*/  int *entity_handles_allocated,
                      /*out*/ int *entity_handles_size, 
                              int *err); 

//  + Boundary iterators:  Given a partition handle, a part handle, and a 
//    neighboring part ID, return an iterator over all entities along
//    the part boundary shared with the neighboring part ID.  

    void prefix_initPartBdryEntIter(iMesh_Instance instance,
                      /*in*/  const prefix_PartitionHandle partition_handle, 
                      /*in*/  const prefix_PartHandle part_handle, 
                      /*in*/  const int requested_entity_type, 
                      /*in*/  const int requested_entity_topology, 
                      /*in*/  const int requested_nbor_part_id, 
                   /*inout*/  iMesh_EntityIterator* entity_iterator, 
                              int* err); 

//   KDD:  Do we need getNextPartBdryEntIter, resetPartBdrEntIter, and
//   KDD:  endPartBdryEntIter?  Or can the getNextEntIter, resetEntIter, and
//   KDD:  endEntIter be used with the iterator produced by initPartBdryEntIter?

    void prefix_initPartBdryEntArrIter(iMesh_Instance instance,
                      /*in*/  const prefix_PartitionHandle partition_handle, 
                      /*in*/  const prefix_PartHandle part_handle, 
                      /*in*/  const int requested_entity_type, 
                      /*in*/  const int requested_entity_topology, 
                      /*in*/  const int requested_array_size, 
                      /*in*/  const int requested_nbor_part_id, 
                   /*inout*/  iMesh_EntityIterator* entity_iterator, 
                              int* err); 

//   KDD: Do we need getNextPartBdryEntArrIter, resetPartBdrEntArrIter, and
//   KDD: endPartBdryEntArrIter?  Or can the getNextEntArrIter, resetEntArrIter,
//   KDD: and endEntArrIter be used with the iterator produced by 
//   KDD: initPartBdryEntArrIter?

////////////////////////////////
//  Add/remove on-process entities to/from on-process part:  Given
//  a partition handle, an entity handle, and a part handle, add/remove the
//  entity to/from the part.  
//  KDD:  If parts don't need their partition handles, this operation can 
//  KDD:  be accomplished by functions that add/remove entities to/from 
//  KDD:  EntitySets.

#define prefix_addEntToPart iMesh_addEntToSet
#define prefix_rmvEntFromPart iMesh_rmvEntFromSet
#define prefix_addEntArrToPart iMesh_addEntArrToSet
#define prefix_rmvEntArrFromPart iMesh_rmvEntArrFromSet


////////////////////////////////
//  Add entities to on-process and/or off-process parts:  This 
//  functionality could be implemented through a send/receive message pair.
//  The receive blocks until all expected data is
//  received; also can update ghost/boundary/internal data as needed.

//  Note:  Some set-up communication may be needed for the receive to
//  determine what messages it is waiting for.  KDD:  Agreed!
//  Note:  Using a pair of calls allows at least some latency hiding.

    void prefix_sendEntArrToParts(iMesh_Instance instance,
                      /*in*/  const prefix_PartitionHandle partition_handle, 
                      /*in*/  const prefix_PartHandle source_part_handle, 
                      /*in*/  const iBase_EntitySetHandle *entity_set_handles,
                      /*in*/  const int entity_handles_size,
                      /*in*/  const int *target_part_ids,
                      /*in*/  int command_code,  // e.g., MIGRATE,COPY
                              int *err);

    void prefix_receiveEntArrToParts(iMesh_Instance instance,
                      /*in*/  const prefix_PartitionHandle partition_handle, 
                      /*in*/  const prefix_PartHandle source_part_handle, 
                   /*inout*/  iBase_EntitySetHandle **entity_set_handles,
                   /*inout*/  int *entity_handles_size,
                     /*out*/  int *entity_handles_allocated,
                   /*inout*/  int *target_part_ids,
                     /*out*/  int *target_part_ids_allocated,
                   /*inout*/  int command_code,  // e.g., MIGRATE,COPY
                              int *err);

////////////////////////////////
//  Provide entity information about a part; e.g., given a partition handle and
//  a part handle, return the number of entities in the part, lists of
//  entities in the part, etc. 

    void prefix_getNumOfType(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
                       const prefix_PartHandle part_handle,
                       const iBase_EntitySetHandle entity_set_handle,
                       const int entity_type, int *num_type, int *err);
    void prefix_getNumOfTopo(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
                       const prefix_PartHandle part_handle,
                       const iBase_EntitySetHandle entity_set_handle,
                       const int entity_topology, int *num_topo, int *err);

    void prefix_getEntSets(iMesh_Instance instance, 
                       const prefix_PartitionHandle partition_handle,
                       const prefix_PartHandle part_handle,
                       const iBase_EntitySetHandle entity_set_handle, 
                       const int num_hops, 
                       iBase_EntitySetHandle** contained_set_handles, 
                       int* contained_set_handles_allocated, 
                       int* contained_set_handles_size, int *err); 

//  KDD:  THE ABOVE FUNCTION WAS NOT ONE OF CARL'S SIXTEEN, BUT IT IS ANALOGOUS 
//  KDD:  TO HIS FUNCTIONS.  DO WE NEED IT?

    void prefix_getAllVtxCoords(iMesh_Instance instance,
                             const prefix_PartitionHandle partition_handle,
                             prefix_PartHandle part_handle,
                             const iBase_EntitySetHandle entity_set_handle,
                             double** coordinates,
                             int* coordinates_allocated,
                             int* coordinates_size,
                             int** in_entity_set,
                             int* in_entity_set_allocated,
                             int* in_entity_set_size,
                             int* storage_order, int *err);

    void prefix_getVtxCoordIndex(iMesh_Instance instance,
                             const prefix_PartitionHandle partition_handle,
                             prefix_PartHandle part_handle,
                             const iBase_EntitySetHandle entity_set_handle,
                             const int requested_entity_type,
                             const int requested_entity_topology,
                             const int entity_adjacency_type,
                             int** offset,
                             int* offset_allocated,
                             int* offset_size,
                             int** index,
                             int* index_allocated,
                             int* index_size,
                             int** entity_topologies,
                             int* entity_topologies_allocated,
                             int* entity_topologies_size, 
                             int *err);

    void prefix_getEntities(iMesh_Instance instance,
                        const prefix_PartitionHandle partition_handle,
                        prefix_PartHandle part_handle,
                        const iBase_EntitySetHandle entity_set_handle,
                        const int entity_type,
                        const int entity_topology,
                        iBase_EntityHandle** entity_handles,
                        int* entity_handles_allocated,
                        int* entity_handles_size,
                        int *err);

    void prefix_getAdjEntities(iMesh_Instance instance,
                           const prefix_PartitionHandle partition_handle,
                           prefix_PartHandle part_handle,
                           const iBase_EntityHandle entity_set_handle,
                           const int entity_type_requestor,
                           const int entity_topology_requestor,
                           const int entity_type_requested,
                           iBase_EntityHandle** adj_entity_handles,
                           int* adj_entity_handles_allocated,
                           int* adj_entity_handles_size,
                           int** offset,
                           int* offset_allocated,
                           int* offset_size,
                           int** in_entity_set,
                           int* in_entity_set_allocated,
                           int* in_entity_set_size, 
                           int *err);

////////////////////////////////
//  Provide entity and array iterators within a part.
    void prefix_initEntIter(iMesh_Instance instance,
                        const prefix_PartitionHandle partition_handle,
                        prefix_PartHandle part_handle,
                        const iBase_EntitySetHandle entity_set_handle,
                        const int requested_entity_type,
                        const int requested_entity_topology,
                        iMesh_EntityIterator* entity_iterator,
                        int *err);
    void prefix_getNextEntIter(iMesh_Instance instance,
                           const prefix_PartitionHandle partition_handle,
                           prefix_PartHandle part_handle,
                           iMesh_EntityIterator entity_iterator,
                           iBase_EntityHandle* entity_handle,
                           int *has_data,
                           int *err);
    void prefix_resetEntIter(iMesh_Instance instance,
                         const prefix_PartitionHandle partition_handle,
                         prefix_PartHandle part_handle,
                         iMesh_EntityIterator entity_iterator,
                         int *err);
    void prefix_endEntIter(iMesh_Instance instance,
                       const prefix_PartitionHandle partition_handle,
                       prefix_PartHandle part_handle,
                       iMesh_EntityIterator entity_iterator,
                       int *err);
    void prefix_initEntArrIter(iMesh_Instance instance,
                           const prefix_PartitionHandle partition_handle,
                           prefix_PartHandle part_handle,
                           const iBase_EntitySetHandle entity_set_handle,
                           const int requested_entity_type,
                           const int requested_entity_topology,
                           const int requested_array_size,
                           iMesh_EntityArrIterator* entArr_iterator,
                           int *err);
    void prefix_getNextEntArrIter(iMesh_Instance instance,
                              const prefix_PartitionHandle partition_handle,
                              prefix_PartHandle part_handle,
                              iMesh_EntityArrIterator entArr_iterator,
                              iBase_EntityHandle** entity_handles,
                              int* entity_handles_allocated,
                              int* entity_handles_size,
                              int *has_data, 
                              int *err);
    void prefix_resetEntArrIter(iMesh_Instance instance,
                            const prefix_PartitionHandle partition_handle,
                            prefix_PartHandle part_handle,
                            iMesh_EntityArrIterator entArr_iterator, 
                            int *err);
    void prefix_endEntArrIter(iMesh_Instance instance,
                          const prefix_PartitionHandle partition_handle,
                          prefix_PartHandle part_handle,
                          iMesh_EntityArrIterator entArr_iterator, 
                          int *err);

//    KDD:  Do we need prefix_getNext...,  prefix_reset...,  and
//    KDD:  prefix_end...?  Or can the iMesh_getNext..., iMesh_reset..., and
//    KDD:  iMesh_end... be used with the iterator produced by prefix_init...?

/*------------------------------------------------------------------------*/
/*------------------------------------------------------------------------*/
/*                           Entity Functionality                         */
/*------------------------------------------------------------------------*/
/*------------------------------------------------------------------------*/

////////////////////////////////
//  Provide part information about an entity: Given an entity and a 
//  partition handle, return the part ID of the part that owns the entity.
//  Must return an error code if the entity is not in the
//  partition (e.g., if the partition assigns surfaces to parts, it
//  doesn't make sense to ask which part owns a given region).

    void prefix_getEntityOwner(iMesh_Instance instance,
                      /*in*/  const prefix_PartitionHandle partition_handle, 
                      /*in*/  iBase_EntityHandle entity_handle,
                     /*out*/  int *part_id,
                              int* err); 
                      
    void prefix_getEntityOwnerArr(iMesh_Instance instance,
                      /*in*/  const prefix_PartitionHandle partition_handle, 
                      /*in*/  iBase_EntityHandle *entity_handles,
                      /*in*/  int entity_handles_size,
                   /*inout*/  int **part_ids,
                   /*inout*/  int *part_ids_allocated,
                              int* err); 
  
////////////////////////////////
//  Provide entity categorization within part (boundary, copy, owned, etc.):
//  + Given a partition handle, a part handle, and an entity handle, return a
//    flag indicating whether the entity is owned by the part or is a copy.

    void prefix_isEntOwner(iMesh_Instance instance,
                    /*in*/ const prefix_PartitionHandle partition_handle, 
                    /*in*/ const prefix_PartHandle part_handle, 
                    /*in*/ const iBase_EntityHandle entity_handle, 
                   /*out*/ int* is_owner, 
                           int *err); 

    void prefix_isEntOwnerArr(iMesh_Instance instance,
                    /*in*/ const prefix_PartitionHandle partition_handle, 
                    /*in*/ const prefix_PartHandle part_handle, 
                    /*in*/ const iBase_EntityHandle *entity_handles, 
                    /*in*/ const int entity_handles_size, 
		 /*inout*/ int** is_owner, 
                 /*inout*/ int* is_owner_allocated, 
                           int *err); 

//  + Given a partition handle, a part handle, and an entity handle, return a
//    flag indicating whether the entity is strictly internal, on a 
//    boundary, or a ghost.

    void prefix_isEntParStatus(iMesh_Instance instance,
                    /*in*/ const prefix_PartitionHandle partition_handle, 
                    /*in*/ const prefix_PartHandle part_handle, 
                    /*in*/ const iBase_EntityHandle entity_handle, 
                   /*out*/ int* par_status, // Values=INTERNAL,BOUNDARY,GHOST
                           int *err); 

    void prefix_isEntParStatusArr(iMesh_Instance instance,
                    /*in*/ const prefix_PartitionHandle partition_handle, 
                    /*in*/ const prefix_PartHandle part_handle, 
                    /*in*/ const iBase_EntityHandle *entity_handles, 
                    /*in*/ const int entity_handles_size, 
		 /*inout*/ int** par_status, // Values=INTERNAL,BOUNDARY,GHOST
                 /*inout*/ int* par_status_allocated, 
                           int *err); 

////////////////////////////////
//  Provide information about copies of entities.  
//  All these functions should work on the local process as well as 
//  remote processes; entity handles returned are likely but not 
//  necessarily remote.  For example, for a valid entity handle, the
//  first function above should return an integer >= 1, and the third
//  function should return at least the owned entity handle.
  
//  +  Given a partition handle, an entity handle and a part ID, return the
//     (remote) entity handle of the entity in the specified part.
//     Return an error if the entity does not exist in the specified part.

     void prefix_getCopyEntOnPart(iMesh_Instance instance,
                    /*in*/ const prefix_PartitionHandle partition_handle, 
                    /*in*/ const iBase_EntityHandle entity_handle, 
                    /*in*/ int* part_id, 
                   /*out*/ iBase_EntityHandle* copy_entity_handle, 
                           int *err); 

//  +  Given a partition handle and an entity handle, return the number 
//     of copies of the entity on processes in the partition's communicator.

     void prefix_getNumCopiesEnt(iMesh_Instance instance,
                    /*in*/ const prefix_PartitionHandle partition_handle, 
                    /*in*/ const iBase_EntityHandle entity_handle, 
                   /*out*/ int *num_copies_ent,
                           int *err); 

//  +  Given a partition handle and an entity handle, return (remote) entity
//     handles and part IDs of all copies of the entity.

     void prefix_getCopiesEnt(iMesh_Instance instance,
                    /*in*/ const prefix_PartitionHandle partition_handle, 
                    /*in*/ const iBase_EntityHandle entity_handle, 
                 /*inout*/ int** part_ids, 
                 /*inout*/ int* part_ids_allocated, 
                   /*out*/ int* part_ids_size, 
                 /*inout*/ iBase_EntityHandle** copies_entity_handles, 
                 /*inout*/ int* copies_entity_handles_allocated, 
                           int *err); 


//  +  Given a partition handle and an entity handle, return the (remote) entity
//     handle and part ID from the owner of the entity (e.g., the 
//     entity handle of the copy that has right-to-modify).

     void prefix_getOwnedEnt(iMesh_Instance instance,
                    /*in*/ const prefix_PartitionHandle partition_handle, 
                    /*in*/ const iBase_EntityHandle entity_handle, 
                   /*out*/ int* owner_part_id, 
                   /*out*/ iBase_EntityHandle* owner_entity_handle, 
                           int *err); 

//  +  Given a partition handle, an entity handle, and a part ID, add/remove
//     copies of the entity to/from the specified part.

     void prefix_addCopyEnt(iMesh_Instance instance,
                    /*in*/ const prefix_PartitionHandle partition_handle, 
                    /*in*/ const iBase_EntityHandle entity_handle, 
                    /*in*/ int target_part_id, 
                           int *err); 

     void prefix_removeCopyEnt(iMesh_Instance instance,
                    /*in*/ const prefix_PartitionHandle partition_handle, 
                    /*in*/ const iBase_EntityHandle entity_handle, 
                    /*in*/ int target_part_id, 
                           int *err); 

/*
------------------------------------------------
Comments and resolved questions:  

- Applications will build partitions by (1) creating a partition handle
  on each process to be included in the partition; (2) adding parts to 
  the partition handle within the process; and (3)
  populating the parts with entities.
- All parts will have global (across the partition's communicator)
  part IDs, but will not have local (to a process) part IDs.  
  Part handles will serve as local IDs.
- For now, we will not include an iterator over local (to a
  process) parts within a partition.  If it is needed, it can be added
  later.
- Functions returning global information (such as the total number 
  of parts in a partition) would have the option of using communication to 
  return their answer and, thus, would have to be called synchronously by 
  users.  We agreed that implementations did not necessarily have to use 
  communication, but the documentation would warn users that 
  communication might occur.
- We will not provide capability to move entire parts to new
  processes;  instead, the new process must create the part in its
  partition handle and then receive (perhaps in bulk) the entities to 
  populate the part.  In other words, parts can be added to only a local 
  partition handle.
- Currently, iMesh doesn't have the functionality to get entities or 
  entity sets by type and tag in serial.  Should it?  
  Many people said it would be useful; others said it could be costly
  (in parallel) or numerically difficult (for floating point values).
  This issue is an iMesh issue, not a parallel interface issue, so
  for this document, the issue is resolved.  The resolution:  If 
  iMesh adopts this capability, we will consider adding it to the
  parallel interface.
  Removed the following global queries; are there analogous serial
  iMesh functions?
  + Total number of entity sets with given type, 
    tag and/or tag name in the partition;
  + Total entities with given type, tag and/or tag name 
    in the partition;
- We will not include functions that return all entities with 
  given characteristics within a partition; the memory use of these
  functions can be large.  Instead, we will return entity information
  with respect to parts and/or mesh instances.  If the user wants such
  info, he should go through the mechanics of gathering it himself so
  that he is painfully aware of how much memory he is allocating.
  Removed the following global queries:
  + All tag names over the partition;
  + All entities in this partition having a given type, tag and/or 
    tag name.
  + All entity sets in this partition having a given
    type, tag and/or tag name.
- We will not include functions that return information about each
  part and/or process in a partition.  Such functions limit memory 
  scalability for large numbers of parts.  If the user wants such
  info, he should go through the mechanics of gathering it himself so
  that he is painfully aware of how much memory he is allocating.
  Removed the following global queries:
  + The number of entities in each part of the partition;
  + The number of entity sets in each part of the partition;
  + The number of entities with given type, tag, and/or
    tag name in each part of the partition;
  + The number of entity sets with given type, tag, 
    and/or tag name in each part of the partition;
  + All tag names in each part of the partition;
- For functions that replace a set handle with a part handle, return
  all appropriate entities in a part, whether they are owned or are 
  copies.  The application can test for ownership if needed.
- We decided that part assignments computed with respect to a set of 
  entities induce part assignments to lower-dimensional entities in an
  implementation-dependent fashion.  That is, if a partition is computed
  with respect to regions, queries about ownership of faces and vertices
  are valid.

------------------------------------------------
Discussed but unresolved questions:

- We discussed but did not yet accept the possibility of adding a
  partition-syncing function that would compute and store global info
  about a partition after modifications of a partition were completed.  We
  discussed allowing this function to compute and store some scalar values
  (e.g., total number of parts in a partition), but ruled out allowing it 
  to compute and store vector data of size O(number of processors) 
  (e.g., the number of parts on each processor).
- Do we need a mechanism to store a mapping of parts to processes in
  files?   This capability could be used in static partitioning to 
  compute on, say, 10 processes a partition with 10K parts that will
  be used on 5K processes.
- We discussed and deferred a decision on adding functions that give
  hints to an implementation about which data mappings the application 
  will use, allowing the implementation to pre-compute them if it chooses 
  to.  The example discussed was mapping between entities and parts, but 
  other examples in iMesh may also exist.
- We discussed and deferred a decision on adding an iterator over entities 
  with given type/topology in a set or part.  We have non-iterator 
  functionality, but not an iterator.  
  KDD:  Is this true?  What is iMesh_initEntIter (and its analogous
  KDD:  prefix_initEntIter)?
- We discussed and deferred a decision on storing in a partition 
  information about which "objects" were used in computing the partition.  
  These objects can be single entities or groups of entities.
  KDD:  Perhaps this capability should be part of the load-balancing service.
- We discussed and deferred a decision on designating a given partition as 
  the "active" partition; i.e., the partition that is actually used in 
  the distribution of mesh data in distributed memory.  We were concerned 
  that when multiple partitions were used, multiple copies of mesh 
  entities would be needed to fully support multiple partitions at the 
  same time.  Designating one partition as "active" would store data 
  with respect to only one partition.

------------------------------------------------
Not-yet-discussed, unresolved questions

Entity questions:
- From Carl:  "getTag*Operate: Again, we haven't got this in serial.  Does 
  the existence of such operations imply that we expect to implement 
  fields as tags? (Because that wasn't what I was assuming about field 
  implementations at all, personally...)  Note that I'm not opposed to 
  this sort of global reduction operation, I just wonder whether it'll see 
  use outside of field-like situations.  If not, then it should be in 
  parallel fields, not parallel mesh, and usage for 
  fields-implemented-as-tags should be handled there."

------------------------------------------------
CVS File Information
   $RCSfile: DraftInterface.h,v $
   $Author: kddevin $
   $Date: 2008/01/22 18:29:30 $
   $Revision: 1.2 $
------------------------------------------------
*/