/* ------------------------------------------------ Background and Terminology: Terminology -- Abstract Data Model: - The term "mesh" refers to an abstraction in the data model; it does not imply a serial or parallel distribution. - The term "partition" refers to an assignment of a set of entities to subsets; like a "mesh," it does not imply a serial or parallel implementation. - An application may use one or more meshes. - Parititions can create subsets of entities from one or more meshes. - Meshes can be subdivided by one or more partitions. - Partitions contain parts. Parts contain the subsets of entities in the partition. Terminology -- Parallelism - A "process" can be thought of as an MPI process. The number of processes can be considered to be the result of MPI_Comm_size. The rank of a process can be thought of as the result of MPI_Comm_rank. We will think in terms of processes rather than processors. Initial implementations of the parallel interface will likely use MPI terminology directly; future implementations may accommodate other communication paradigms and libraries. - Partitions have communicators associated with them. These communicators can be thought of as MPI communicators. - "Global" operations are operations performed with respect to a partition's communicator. - "Local" operations are operations performed with respect to a part or a mesh instance within a process. - Part A "neighbors" Part B if A and B share entities of a given dimension; functions computing part neighbors will accept an entity type (possibly iBase_ALL_TYPES). Terminology -- Interfaces - Each process has one or more "mesh instances." A mesh instance can be thought of as a mesh database. An implementation should support the existence of more than one mesh instance per process (e.g., it should always associate mesh data with a mesh instance). However, we expect applications would most often use only one mesh instance per process. - There is one root set per mesh instance. - Each process may have one or more partition handles. - A partition partitions entities in one mesh instance. - Entities in a mesh instance can be partitioned by one or more partitions. Mesh instances know which partitions they contain. - Parts can be accessed by either part IDs (integers) or part handles. Part IDs are valid globally; part handles are local to a process. Part IDs are NOT assumed to be consecutive; nor are they assumed to be in the range [0:numParts-1]. Thus, processes can generate new Part IDs without communication with other processes. TODO: AN IMPLEMENTATION IDEA (BUT NOT A REQUIREMENT) FOR PART IDS FROM CARL: TODO: Using n bits for process rank, TODO: m bits for local ID would create part ID's that were easy TODO: to create and interpret... with cleverness, we could work things so TODO: that n+m <= 31 (a billion part ID's...) so that this all looks like an TODO: int, and we don't have to mess with the sign bit. TODO: In not assuming the range [0:numParts-1] for part IDs, we give up TODO: the easy ability to use part IDs as indices (as, for example, Onkar TODO: did in his Entity Level comments of Feb 5). TODO: SHOULD PART IDS BE OF TYPE prefix_PartID_type? Which leads me to TODO: my next question.... TODO: ARE PART IDS REALLY REMOTE PART HANDLES, THEN? SHOULD WE MAKE A TODO: DISTINCTION BETWEEN PART IDS AND PART HANDLES? THE INTEGER VALUES TODO: HAD VALUE IF THEY COULD BE USED AS ARRAY INDICES, BUT SINCE THEY TODO: CANNOT, PERHAPS WE SHOULD JUST CALL THEM REMOTE PART HANDLES AND TODO: REMOVE THE DISTINCTION BETWEEN IDS AND HANDLES. WE COULD THEN TODO: REMOVE FUNCTIONS MAPPING BETWEEN IDS AND HANDLES, BUT WE WOULD TODO: PROBABLY NEED TO ADD A TEST FOR WHETHER A PART HANDLE WAS REMOTE TODO: OR LOCAL. - TODO: LORI OBSERVES: IN MANY FUNCTIONS THERE'S A "getNumX" AND A "getX". TODO: WE SHOULD LOOK AT WHETHER OR NOT WE ALWAYS NEED BOTH FUNCTIONS TODO: OR IF WE CAN JUST USE getX, WHICH RETURNS THE NUM AS A BYPRODUCT. TODO: I REALIZE THERE ARE MANY CASES WHERE YOU DO NEED TO SEPARATE THE TODO: CALLS, BUT PERHAPS NOT ALL. TODO: KDD ADDS: WHAT IS THE CONVENTION FOR getNumX AND getX IN ITAPS? TODO: PROVIDING getNumX ALLOWS THE APPLICATION TO PREALLOCATE MEMORY. - TODO: NEED TO DECIDE WHICH FUNCTIONS NEED both getX AND getXArr VERSIONS. Assumptions: - Each part is wholly contained within a process. - A process may have multiple parts. - Many iMesh functions that accept EntitySet handles are also useful in the context of part handles. These functions will be reinterpreted so that they can accept either an EntitySet handle, or a part handle. See Carl's Oct 15 email for more detail. - Entities in partitions can have unique global IDs. Uniqueness of global IDs is managed at the partition level. Generation and management of global IDs will be provided as a service above the parallel iMesh interface. Operations to manage global IDs of entities that are partitioned include set/retrieve global IDs; generate global IDs; set/retrieve global ID size; map between entities and global IDs; compare global IDs. Since there was little disagreement with Vitus'/Karen's document on global IDs, we will use it in the first draft with modifications to associate the "global" nature of the IDs with partitions. */ /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ /* Partition Functionality */ /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ //////////////////////////////// // Create a partition handle: Given a mesh instance to contain // the partition and a communicator, return a partition // handle. May have different creation routines for different // communication systems; once created, the application shouldn't have to // worry about the communication system again. // For serial use, *communicator may be MPI_COMM_SELF or communicator may // be NULL. // TODO: May still revise this function to make it less MPI specific. // TODO: Don't want to require applications to link with MPI if the // TODO: implementation doesn't require it. // TODO: May define an ITAPS_Comm name typedef'ed appropriately. void prefix_createPartitionMPI(iMesh_Instance instance, /*in*/ MPI_Comm *communicator, /*out*/ prefix_PartitionHandle *partition_handle, int *err); //////////////////////////////// // Destroy a partition handle: Given a partition handle, destroy it and set // it to NULL. void prefix_destroyPartition(iMesh_Instance instance, /*inout*/ prefix_PartitionHandle *partition_handle, int *err); //////////////////////////////// // Update a partition after parts have been added. // Gives the implementation an opportunity to locally store info // about the partition so that queries on the partition can be // performed without synchronous communication. // TODO: Determine which values must be updated to allow other // TODO: interface functions to be performed without synchronous // TODO: communication. void prefix_updatePartitionPar(iMesh_Instance instance, prefix_PartitionHandle partition_handle, int *err); //////////////////////////////// // Given a partition handle, return its communicator. void prefix_getPartitionCommMPI(iMesh_Instance instance, /*in*/ prefix_PartitionHandle partition_handle, /*out*/ MPI_Comm *communicator, int *err); //////////////////////////////// // Given a mesh instance, return number of partition handles and // all partition handles associated with the mesh instance. void prefix_getNumPartitions(iMesh_Instance instance, /*out*/ int *num_partitions, int *err); void prefix_getPartitions(iMesh_Instance instance, /*inout*/ prefix_PartitionHandle **partition_handle, /*inout*/ int *partition_handle_allocated, /*out*/ int *partition_handle_size, int *err); //////////////////////////////// // Given a partition handle, return the total global number of parts // in the partition. // TODO: Can this function be made local (through updates from // TODO: prefix_updatePartitionPar)? void prefix_getNumPartsPar(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, int *num_global_part, int *err); //////////////////////////////// // Map from parts to processes: // Given a partition handle and a part ID, return the rank of the // process that owns the part. // TODO: Are these functions required to be local (through updates from // TODO: prefix_updatePartitionPar)? void prefix_getPartRank(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int part_id, /*out*/ int *rank, int *err); void prefix_getPartRankArr(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int *part_id, /*in*/ const int part_id_size, /*inout*/ int **rank, /*inout*/ int *rank_allocated, int *err); //////////////////////////////// // Map from processes to parts: // + Given a partition handle and a process rank, // return the number of parts owned by the process. // TODO: Are these functions required to be local (through updates from // TODO: prefix_updatePartitionPar)? void prefix_getNumParts(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int rank, /*out*/ int *num_parts, int *err); void prefix_getNumPartsArr(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int *rank, /*in*/ const int rank_size, /*inout*/ int **num_parts, /*inout*/ int *num_parts_allocated, int *err); // + Given a partition handle and a process rank, // return the part IDs owned by the process. // TODO: Are these functions required to be local (through updates from // TODO: prefix_updatePartitionPar)? void prefix_getPartIds(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int rank, /*inout*/ int **pids, /*inout*/ int *pids_allocated, /*out*/ int *pids_size, int *err); void prefix_getPartIdsArr(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int *rank, /*in*/ const int rank_size, /*inout*/ int **pids, /*inout*/ int *pids_allocated, /*out*/ int *pids_size, int *err); //////////////////////////////// // Provide global mesh information about a partition. // Note that these functions may require communication and, thus, // would have to be called by all processes in the partition handle. // Given a mesh instance and partition handle, return: // + Total number of entities with given type or topology in the partition. // TODO: TIM ASKS: DOES IT MAKE SENSE TO HAVE AN ENTITY SET HANDLE PARAMEMTER // TODO: IN THESE FUNCTIONS? DO ENTITY SETS GIVEN ON EACH PROCESS HAVE TO BE // TODO: ONES SHARED IN PARALLEL? // TODO: KDD SAYS: YES, THIS FUNCTION IS ANALOGOUS TO iMesh_getNumOfType, // TODO: WHICH TAKES THE ROOT SET OR ANY OTHER ENTITY SET. // TODO: KDD ADDS: WE AGREED THAT ENTITY SETS COULD SPAN ACROSS PARTS AND // TODO: PROCESSES. IF ENTITY SET HANDLES ARE, SAY, ADDRESSES, HOW DOES THE // TODO: IMPLEMENTATION KNOW THAT ENTITY SET HANDLES ON DIFFERENT PROCESSES // TODO: ACTUALLY ARE THE SAME ENTITY SET? void prefix_getNumOfTypePar(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, const iBase_EntitySetHandle entity_set_handle, const int entity_type, int *num_type, int *err); void prefix_getNumOfTopoPar(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, const iBase_EntitySetHandle entity_set_handle, const int entity_topology, int *num_topo, int *err); // + Total number of entity sets in the partition. // // void prefix_getNumEntSetsPar(iMesh_Instance instance, // const prefix_PartitionHandle partition_handle, // const iBase_EntitySetHandle entity_set_handle, // const int num_hops, int *num_sets, int *err); // // TODO: THIS FUNCTION WAS NOT ONE OF CARL'S SIXTEEN, BUT IT IS ANALOGOUS TO // TODO: HIS FUNCTIONS. DO WE NEED IT? // TODO: TIM SAYS "NO"; ENTITY SETS AREN'T IN PARTITIONS, SO THIS FUNCTION IS // TODO: POORLY DEFINED. I AGREE, AS I COULDN'T RIGHT AN APPROPRIATE COMMENT // TODO: DESCRIBING THE FUNCTION. SO I'LL COMMENT IT OUT FOR NOW (AND DELETE // TODO: IT LATER IF THERE ARE NO OBJECTIONS). /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ /* Part Functionality */ /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ //////////////////////////////// // Given a partition handle, create a part and add it to the // partition on the process invoking the creation. Return part handle. void prefix_createPart(iMesh_Instance instance, /*in*/ prefix_PartitionHandle partition_handle, /*out*/ prefix_PartHandle *part_handle, int *err); //////////////////////////////// // Given a partition handle and a part handle, remove the part // from the partition, destroy the part, and invalidate the part handle. void prefix_destroyPart(iMesh_Instance instance, /*in*/ prefix_PartitionHandle partition_handle, /*inout*/ prefix_PartHandle *part_handle, int *err); //////////////////////////////// // Map between part handles and part IDs. // + Given a partition handle and a part handle, return the part ID. // Note that part handles are local as described above. // TODO: SEE DISCUSSION OF PART IDS IN INITIAL COMMENTS. // TODO: MAY NOT NEED THESE FUNCTIONS. void prefix_getPartIdFromPartHandle(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*out*/ int *part_id, int *err); void prefix_getPartIdsFromPartHandlesArr(iMesh_Instance instance, /*in*/ const prefix_PartHandle *part_handles, /*in*/ const int part_handles_size, /*inout*/ int **part_ids, /*inout*/ int *part_ids_allocated, int *err); // + Given a partition handle and a part ID, return the part handle // if on process; return error code otherwise. // TODO: SEE DISCUSSION OF PART IDS IN INITIAL COMMENTS. // TODO: MAY NOT NEED THESE FUNCTIONS; MAY REPLACE THEM WITH TEST FOR // TODO: LOCAL OR REMOTE PART HANDLE. void prefix_getPartHandleFromPartId(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ int part_id, /*out*/ prefix_PartHandle *part_handle, int *err); void prefix_getPartHandlesFromPartIdsArr(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int *part_ids, /*in*/ const int part_ids_size, /*inout*/ prefix_PartHandle **part_handles, /*inout*/ int *part_handles_allocated, int *err); //////////////////////////////// // Identify parts that neighbor a given part. // TODO: NEED NEW DEFINITION OF PART NEIGHBOR. // TODO: ARE THESE FUNCTIONS REQUIRED TO BE LOCAL (THROUGH UPDATES FROM // TODO: prefix_updatePartitionPar)? // + Given a partition handle, a part handle, and an entity type, // return the number of parts in the partition that neighbor the given part // (using the definition of "neighbor" above). void prefix_getNumPartNbors(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const prefix_PartHandle part_handle, /*in*/ int entity_type, /*out*/ int *num_part_nbors, int *err); void prefix_getNumPartNborsArr(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const prefix_PartHandle *part_handles, /*in*/ const int part_handles_size, /*in*/ int entity_type, /*inout*/ int **num_part_nbors, /*inout*/ int *num_part_nbors_allocated, int *err); // + Given a partition handle and a part handle, return the part IDs of all // parts in the partition that neighbor the given part. void prefix_getPartNbors(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const prefix_PartHandle part_handle, /*in*/ int entity_type, /*inout*/ int **part_nbor_ids, /*inout*/ int *part_nbor_ids_allocated, /*out*/ int *part_nbor_ids_size, int *err); void prefix_getPartNborsArr(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const prefix_PartHandle *part_handles, /*in*/ const int part_handles_size, /*in*/ int entity_type, /*inout*/ int **part_nbor_ids, /*inout*/ int *part_nbor_ids_allocated, /*out*/ int *part_nbor_ids_size, int *err); //////////////////////////////// // Provide part boundary info: // Note: Allow optional specification of desired entity types and // topologies; allow neighboring part ID = -1 to count/include all qualifying // boundary entities of part. // + Given a partition handle, a part handle, entity type and topology, and a // neighboring part ID, return the number of qualifying entities on // the part boundary shared with the neighboring part ID. void prefix_getNumPartBdryEnts(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const int requested_entity_type, /*in*/ const int requested_entity_topology, /*in*/ const int requested_nbor_part_id, /*out*/ int *num_nbor_entities, int *err); // + Given a partition handle, a part handle, entity type and topology, and a // neighboring part ID, return an array of entity handles for all qualifying // entities along the part boundary shared with the neighboring part ID. void prefix_getPartBdryEnts(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const int requested_entity_type, /*in*/ const int requested_entity_topology, /*in*/ const int requested_nbor_part_id, /*inout*/ iBase_EntityHandle **entity_handles, /*inout*/ int *entity_handles_allocated, /*out*/ int *entity_handles_size, int *err); // + Boundary iterators: Given a partition handle, a part handle, and a // neighboring part ID, return an iterator over all entities along // the part boundary shared with the neighboring part ID. // Functionality for getNext, reset, and end is // provided through the regular iMesh iterator functions. void prefix_initPartBdryEntIter(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const int requested_entity_type, /*in*/ const int requested_entity_topology, /*in*/ const int requested_nbor_part_id, /*inout*/ iMesh_EntityIterator* entity_iterator, int* err); void prefix_initPartBdryEntArrIter(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const int requested_entity_type, /*in*/ const int requested_entity_topology, /*in*/ const int requested_array_size, /*in*/ const int requested_nbor_part_id, /*inout*/ iMesh_EntityIterator* entity_iterator, int* err); //////////////////////////////// // Add/remove on-process entities to/from on-process part: Given // a partition handle, an entity handle, and a part handle, add/remove the // entity to/from the part. // On-process add/remove can be accomplished by functions // that add/remove entities to/from EntitySets. // TODO: ONKAR SAYS #define CAN BE PROBLEMATIC FOR FUNCTION NAMES. // TODO: WE'LL WORK OUT THE RIGHT MECHANISM. #define prefix_addEntToPart iMesh_addEntToSet #define prefix_rmvEntFromPart iMesh_rmvEntFromSet #define prefix_addEntArrToPart iMesh_addEntArrToSet #define prefix_rmvEntArrFromPart iMesh_rmvEntArrFromSet //////////////////////////////// // Add entities to on-process and/or off-process parts: This // functionality could be implemented through a send/receive message pair. // The receive blocks until all expected data is // received; also can update ghost/boundary/internal data as needed. // The implementation has everything it needs to determine sends and // receives. If desired, it can send and post non-blocking receives. // Upon return, the application can do other stuff for latency hiding. void prefix_sendEntArrToPartsPar(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle *entity_handles, /*in*/ const int entity_handles_size, /*in*/ const int *target_part_ids, /*in*/ int command_code, // e.g., MIGRATE,COPY /*in*/ int update_ghost, // e.g., YES,NO /*in*/ int message_tag, int *err); // The implementation can post blocking receives or wait for non- // blocking receives previously posted. Targets are correct, because // there may be multiple parts on a processor and the application does // not need to know sources. The message_tag ties // a send and receive pair together. void prefix_recvEntArrToPartsPar(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*inout*/ iBase_EntityHandle **entity_handles, /*inout*/ int *entity_handles_size, /*out*/ int *entity_handles_allocated, /*inout*/ int *target_part_ids, /*out*/ int *target_part_ids_allocated, /*inout*/ int command_code, // e.g., MIGRATE,COPY /*in*/ int update_ghost, // e.g., YES,NO /*in*/ int message_tag, int *err); //////////////////////////////// // Provide entity information about a part. // Given an entity set handle // and a part handle, return the number of entities of given type/topo // that are in both the part and the entity set. void prefix_getNumOfType(iMesh_Instance instance, const prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int entity_type, int *num_type, int *err); void prefix_getNumOfTopo(iMesh_Instance instance, const prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int entity_topology, int *num_topo, int *err); // void prefix_getEntSets(iMesh_Instance instance, // const prefix_PartHandle part_handle, // const iBase_EntitySetHandle entity_set_handle, // const int num_hops, // iBase_EntitySetHandle** contained_set_handles, // int* contained_set_handles_allocated, // int* contained_set_handles_size, int *err); // // TODO: THE ABOVE FUNCTION WAS NOT ONE OF CARL'S SIXTEEN, BUT IT IS ANALOGOUS // TODO: TO HIS FUNCTIONS. DO WE NEED IT? // TODO: TIM SAYS "NO"; ENTITY SETS AREN'T IN PARTITIONS, SO THIS FUNCTION IS // TODO: POORLY DEFINED. I AGREE, AS I COULDN'T RIGHT AN APPROPRIATE COMMENT // TODO: DESCRIBING THE FUNCTION. SO I'LL COMMENT IT OUT FOR NOW (AND DELETE // TODO: IT LATER IF THERE ARE NO OBJECTIONS). // Given an entity set handle // and a part handle, return vertex information for vertices // that are in both the part and the entity set. void prefix_getAllVtxCoords(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, double** coordinates, int* coordinates_allocated, int* coordinates_size, int** in_entity_set, int* in_entity_set_allocated, int* in_entity_set_size, int* storage_order, int *err); void prefix_getVtxCoordIndex(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int requested_entity_type, const int requested_entity_topology, const int entity_adjacency_type, int** offset, int* offset_allocated, int* offset_size, int** index, int* index_allocated, int* index_size, int** entity_topologies, int* entity_topologies_allocated, int* entity_topologies_size, int *err); // Given an entity set handle // and a part handle, return entity handles for entities // that are in both the part and the entity set. void prefix_getEntities(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int entity_type, const int entity_topology, iBase_EntityHandle** entity_handles, int* entity_handles_allocated, int* entity_handles_size, int *err); void prefix_getAdjEntities(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntityHandle entity_set_handle, const int entity_type_requestor, const int entity_topology_requestor, const int entity_type_requested, iBase_EntityHandle** adj_entity_handles, int* adj_entity_handles_allocated, int* adj_entity_handles_size, int** offset, int* offset_allocated, int* offset_size, int** in_entity_set, int* in_entity_set_allocated, int* in_entity_set_size, int *err); //////////////////////////////// // Provide entity and array iterators for entities within both a given part // and a given entity set. // Functionality for getNext, reset, and end is // provided through the regular iMesh iterator functions. void prefix_initEntIter(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int requested_entity_type, const int requested_entity_topology, iMesh_EntityIterator* entity_iterator, int *err); void prefix_initEntArrIter(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int requested_entity_type, const int requested_entity_topology, const int requested_array_size, iMesh_EntityArrIterator* entArr_iterator, int *err); /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ /* Entity Functionality */ /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ //////////////////////////////// // Provide part information about an entity: Given an entity and a // partition handle, return the part ID of the part that owns the entity. // Return an error code if an entity is not in the partition. // TODO: SHOULD MORE PRECISELY DEFINE WHEN ERROR CODE IS RETURNED. // TODO: E.G., WHEN AN ENTITY IS NOT OWNED BY ANY PROCESSES IN THE // TODO: PARTITIONS COMMUNICATOR; WHEN AN ENTITY IS NOT "CONNECTED" TO // TODO: ANY ENTITIES USED TO GENERATE THE PARTITION. void prefix_getEntOwnerPartId(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ iBase_EntityHandle entity_handle, /*out*/ int *part_id, int* err); void prefix_getEntOwnerPartIdArr(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ iBase_EntityHandle *entity_handles, /*in*/ int entity_handles_size, /*inout*/ int **part_ids, /*inout*/ int *part_ids_allocated, int* err); //////////////////////////////// // Provide entity categorization within part. // + Given a partition handle, a part handle, and an entity handle, return a // flag indicating whether the entity is owned by the part. void prefix_isEntOwner(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const iBase_EntityHandle entity_handle, /*out*/ int* is_owner, int *err); void prefix_isEntOwnerArr(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const iBase_EntityHandle *entity_handles, /*in*/ const int entity_handles_size, /*inout*/ int** is_owner, /*inout*/ int* is_owner_allocated, int *err); // + Given a partition handle, a part handle, and an entity handle, return a // flag indicating whether the entity is strictly internal, on a // boundary, or a ghost. void prefix_isEntParStatus(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const iBase_EntityHandle entity_handle, /*out*/ int* par_status, // Values=INTERNAL,BOUNDARY,GHOST int *err); void prefix_isEntParStatusArr(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const iBase_EntityHandle *entity_handles, /*in*/ const int entity_handles_size, /*inout*/ int** par_status, // Values=INTERNAL,BOUNDARY,GHOST /*inout*/ int* par_status_allocated, int *err); //////////////////////////////// // Provide information about copies of entities. // All these functions should work on the local process as well as // remote processes; entity handles returned are likely but not // necessarily remote. For example, for a valid entity handle, the // first function above should return an integer >= 1, and the third // function should return at least the owned entity handle. // TODO: ONKAR SAYS THESE FUNCTIONS DO NOT REQUIRE COMMUNICATION, BUT // TODO: FOR MANY OF THEM, I DON'T UNDERSTAND HOW THEY DON'T. // + Given a partition handle, an entity handle and a part ID, return the // (remote) entity handle of the entity in the specified part. // Return an error if the entity does not exist in the specified part. // TODO: LORI OBSERVES: I STRUGGLE WITH THE copyEntOnPart FUNCTIONS... I // TODO: DON'T FULLY UNDERSTAND WHAT THIS ADDS TO THE INTERFACE AND i THINK IT // TODO: COULD LEAD TO HEADACHES WHEN THINKING ABOUT CONSISTENCY OF // TODO: INFORMATION ACROSS PROCESSORS. void prefix_getCopyEntOnPart(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*in*/ int* part_id, /*out*/ iBase_EntityHandle* copy_entity_handle, int *err); // + Given a partition handle and an entity handle, return the number // of copies of the entity in the partition. void prefix_getNumCopiesEnt(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*out*/ int *num_copies_ent, int *err); // + Given a partition handle and an entity handle, return the part IDs // of copies of the entity in the partition. (Requested by Onkar.) void prefix_getCopiesEntPartIDs(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*inout*/ int** part_ids, /*inout*/ int* part_ids_allocated, /*out*/ int* part_ids_size, int *err); // + Given a partition handle and an entity handle, return (remote) entity // handles and part IDs of all copies of the entity. void prefix_getCopiesEnt(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*inout*/ int** part_ids, /*inout*/ int* part_ids_allocated, /*out*/ int* part_ids_size, /*inout*/ iBase_EntityHandle** copies_entity_handles, /*inout*/ int* copies_entity_handles_allocated, int *err); // + Given a partition handle and an entity handle, return the (remote) entity // handle and part ID from the owner of the entity (e.g., the // entity handle of the copy that has right-to-modify). void prefix_getOwnedEnt(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*out*/ int* owner_part_id, /*out*/ iBase_EntityHandle* owner_entity_handle, int *err); // + Given a partition handle, an entity handle, and a part ID, add/remove // copies of the entity to/from the specified part. // TODO: TIM ASKS WHETHER THESE FUNCTIONS COMMUNICATE. I HAD ASSUMED // TODO: THEY DID, SINCE THEY TOOK PARTITION HANDLES AND TARGET PART IDS // TODO: AS ARGUMENTS. BUT THEN THEY WOULD HAVE TO BE CALLED // TODO: SYNCHRONOUSLY. HOW WOULD THAT WORK? AND HOW WOULD THEY DIFFER // TODO: FROM prefix_sendEntArrToPartsPar/prefix_recvEntArrToPartsPar? // TODO: LORI OBSERVES: SEEMS AS THOUGH addCopyEnt/rmvCopyEnt COULD BE // TODO: COMBINED WITH THE OTHER ADD FUNCTIONS (THAT ARE LIKE ENTITYSET // TODO: FUNCTIONS) WITH AN APPROPRIATE ENUMERATOR (e.g. OWNED, COPY). // TODO: I THINK THAT ASSUMES THE FUNCTIONS WORK LOCALLY. // TODO: THERE IS TOO MUCH CONFUSION ON THESE FUNCTIONS // TODO: FOR ME TO MAKE CHANGES. WE'LL DISCUSS THEM IN A PHONE CONF. // TODO: WE NEED TO DETERMINE WHAT CAPABILITY WE NEED. void prefix_addCopyEnt(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*in*/ int target_part_id, int *err); void prefix_rmvCopyEnt(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*in*/ int target_part_id, int *err); /* ------------------------------------------------ Comments and resolved questions: - Applications will build partitions by (1) creating a partition handle on each process to be included in the partition; (2) adding parts to the partition handle within the process; and (3) populating the parts with entities. - All parts will have global (across the partition's communicator) part IDs, but will not have local (to a process) part IDs. Part handles will serve as local IDs. - For now, we will not include an iterator over local (to a process) parts within a partition. If it is needed, it can be added later. - Functions returning global information (such as the total number of parts in a partition) would have the option of using communication to return their answer and, thus, would have to be called synchronously by users. We agreed that implementations did not necessarily have to use communication, but the documentation would warn users that communication might occur. - We will not provide capability to move entire parts to new processes; instead, the new process must create the part in its partition handle and then receive (perhaps in bulk) the entities to populate the part. In other words, parts can be added to only a local partition handle. - Currently, iMesh doesn't have the functionality to get entities or entity sets by type and tag in serial. Should it? Many people said it would be useful; others said it could be costly (in parallel) or numerically difficult (for floating point values). This issue is an iMesh issue, not a parallel interface issue, so for this document, the issue is resolved. The resolution: If iMesh adopts this capability, we will consider adding it to the parallel interface. Removed the following global queries; are there analogous serial iMesh functions? + Total number of entity sets with given type, tag and/or tag name in the partition; + Total entities with given type, tag and/or tag name in the partition; - We will not include functions that return all entities with given characteristics within a partition; the memory use of these functions can be large. Instead, we will return entity information with respect to parts and/or mesh instances. If the user wants such info, he should go through the mechanics of gathering it himself so that he is painfully aware of how much memory he is allocating. Removed the following global queries: + All tag names over the partition; + All entities in this partition having a given type, tag and/or tag name. + All entity sets in this partition having a given type, tag and/or tag name. - We will not include functions that return information about each part and/or process in a partition. Such functions limit memory scalability for large numbers of parts. If the user wants such info, he should go through the mechanics of gathering it himself so that he is painfully aware of how much memory he is allocating. Removed the following global queries: + The number of entities in each part of the partition; + The number of entity sets in each part of the partition; + The number of entities with given type, tag, and/or tag name in each part of the partition; + The number of entity sets with given type, tag, and/or tag name in each part of the partition; + All tag names in each part of the partition; - For functions that replace a set handle with a part handle, return all appropriate entities in a part, whether they are owned or are copies. The application can test for ownership if needed. - We decided that part assignments computed with respect to a set of entities induce part assignments to lower-dimensional entities in an implementation-dependent fashion. That is, if a partition is computed with respect to regions, queries about ownership of faces and vertices are valid. ------------------------------------------------ Discussed but unresolved questions: - We discussed but did not yet accept the possibility of adding a partition-syncing function that would compute and store global info about a partition after modifications of a partition were completed. We discussed allowing this function to compute and store some scalar values (e.g., total number of parts in a partition), but ruled out allowing it to compute and store vector data of size O(number of processors) (e.g., the number of parts on each processor). - Do we need a mechanism to store a mapping of parts to processes in files? This capability could be used in static partitioning to compute on, say, 10 processes a partition with 10K parts that will be used on 5K processes. - We discussed and deferred a decision on adding functions that give hints to an implementation about which data mappings the application will use, allowing the implementation to pre-compute them if it chooses to. The example discussed was mapping between entities and parts, but other examples in iMesh may also exist. - We discussed and deferred a decision on adding an iterator over entities with given type/topology in a set or part. We have non-iterator functionality, but not an iterator. KDD: Is this true? What is iMesh_initEntIter (and its analogous KDD: prefix_initEntIter)? - We discussed and deferred a decision on storing in a partition information about which "objects" were used in computing the partition. These objects can be single entities or groups of entities. KDD: Perhaps this capability should be part of the load-balancing service. - We discussed and deferred a decision on designating a given partition as the "active" partition; i.e., the partition that is actually used in the distribution of mesh data in distributed memory. We were concerned that when multiple partitions were used, multiple copies of mesh entities would be needed to fully support multiple partitions at the same time. Designating one partition as "active" would store data with respect to only one partition. ------------------------------------------------ Not-yet-discussed, unresolved questions Entity questions: - From Carl: "getTag*Operate: Again, we haven't got this in serial. Does the existence of such operations imply that we expect to implement fields as tags? (Because that wasn't what I was assuming about field implementations at all, personally...) Note that I'm not opposed to this sort of global reduction operation, I just wonder whether it'll see use outside of field-like situations. If not, then it should be in parallel fields, not parallel mesh, and usage for fields-implemented-as-tags should be handled there." Ghosts and copies: - From Tim: Need functionality to exchange tag data for entities on interfaces and ghost entities. ------------------------------------------------ CVS File Information $RCSfile: DraftInterface.h,v $ $Author: kddevin $ $Date: 2008/02/11 21:19:16 $ $Revision: 1.7 $ ------------------------------------------------ */