/* ------------------------------------------------ Background and Terminology: Terminology -- Abstract Data Model: - The term "mesh" refers to an abstraction in the data model; it does not imply a serial or parallel distribution. - The term "partition" refers to an assignment of a set of entities to subsets; like a "mesh," it does not imply a serial or parallel implementation. - An application may use one or more meshes. - Parititions can create subsets of entities from one or more meshes. - Meshes can be subdivided by one or more partitions. - Partitions contain parts. Parts contain the subsets of entities in the partition. Terminology -- Parallelism - A "process" can be thought of as an MPI process. The number of processes can be considered to be the result of MPI_Comm_size. The rank of a process can be thought of as the result of MPI_Comm_rank. We will think in terms of processes rather than processors. Initial implementations of the parallel interface will likely use MPI terminology directly; future implementations may accommodate other communication paradigms and libraries. - Partitions have communicators associated with them. These communicators can be thought of as MPI communicators. - "Global" operations are operations performed with respect to a partition's communicator. - "Local" operations are operations performed with respect to a part or a mesh instance within a process. - Part A "neighbors" Part B if A and B share entities of a given dimension; functions computing part neighbors will accept an entity type (possibly iBase_ALL_TYPES). TODO: NEED NEW DEFINITION OF PART NEIGHBOR. - Each function description includes its communication requirements. The options are described here: + COMMUNICATION: Collective -- the function must be called by all processes in the partition's communicator. + COMMUNICATION: Point-to-Point -- communication is used, but the communication is from one process to only one other process. The receiving process must issue an appropriate receive call to receive the message. + COMMUNICATION: None -- the function does not use communication; only local operations are performed. + COMMUNICATION: None++ -- no communication is done ASSUMING THE VALUES ARE PRECOMPUTED BY prefix_UpdatePartitionPar. Once we accept a list of values to be updated, we'll change the "None++" appropriately. + COMMUNICATION: ??? -- we still need to decide. Terminology -- Interfaces - Each process has one or more "mesh instances." A mesh instance can be thought of as a mesh database. An implementation should support the existence of more than one mesh instance per process (e.g., it should always associate mesh data with a mesh instance). However, we expect applications would most often use only one mesh instance per process. - There is one root set per mesh instance. - Each process may have one or more partition handles. - A partition partitions entities in one mesh instance. - Entities in a mesh instance can be partitioned by one or more partitions. Mesh instances know which partitions they contain. - Parts can be accessed by part handles. Most functions accepting part handles operate correctly on only local part handles (part handles for parts on the calling process); they will return an error for remote (off-process) part handles. A few exceptions exist to return process ranks for remote parts, return neighboring part info, etc. In these cases, remote part handles are acceptable arguments. AN IMPLEMENTATION IDEA (BUT NOT A REQUIREMENT) FOR PART HANDLES FROM CARL: Using n bits for process rank, m bits for local part number would create part handles that were easy to create and interpret... with cleverness, one could work things so that n+m <= 31 (a billion part handless...) so that this all looks like an int. - TODO: LORI OBSERVES: IN MANY FUNCTIONS THERE'S A "getNumX" AND A "getX". TODO: WE SHOULD LOOK AT WHETHER OR NOT WE ALWAYS NEED BOTH FUNCTIONS TODO: OR IF WE CAN JUST USE getX, WHICH RETURNS THE NUM AS A BYPRODUCT. TODO: I REALIZE THERE ARE MANY CASES WHERE YOU DO NEED TO SEPARATE THE TODO: CALLS, BUT PERHAPS NOT ALL. TODO: KDD ADDS: WHAT IS THE CONVENTION FOR getNumX AND getX IN ITAPS? TODO: PROVIDING getNumX ALLOWS THE APPLICATION TO PREALLOCATE MEMORY. - TODO: NEED TO DECIDE WHICH FUNCTIONS NEED both getX AND getXArr VERSIONS. Assumptions: - Each part is wholly contained within a process. - A process may have multiple parts. - Many iMesh functions that accept EntitySet handles are also useful in the context of part handles. These functions will be reinterpreted so that they can accept either an EntitySet handle, or a part handle. See Carl's Oct 15 email for more detail. - Generation and management of global IDs for entities will be provided as a service above the parallel iMesh interface. Uniqueness of global IDs is managed at the partition level. */ /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ /* Partition Functionality */ /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ //////////////////////////////// // Create a partition handle: Given a mesh instance to contain // the partition and a communicator, return a partition // handle. May have different creation routines for different // communication systems; once created, the application shouldn't have to // worry about the communication system again. // For serial use, *communicator may be MPI_COMM_SELF or communicator may // be NULL. // TODO: May still revise this function to make it less MPI specific. // TODO: Don't want to require applications to link with MPI if the // TODO: implementation doesn't require it. // TODO: May define an ITAPS_Comm name typedef'ed appropriately. // COMMUNICATION: Collective. void prefix_createPartitionPar(iMesh_Instance instance, /*in*/ MPI_Comm *communicator, /*out*/ prefix_PartitionHandle *partition_handle, int *err); //////////////////////////////// // Destroy a partition handle: Given a partition handle, destroy it and set // it to NULL. // COMMUNICATION: Collective. void prefix_destroyPartitionPar(iMesh_Instance instance, /*inout*/ prefix_PartitionHandle *partition_handle, int *err); //////////////////////////////// // Given a partition handle, return its communicator. // COMMUNICATION: None void prefix_getPartitionComm(iMesh_Instance instance, /*in*/ prefix_PartitionHandle partition_handle, /*out*/ MPI_Comm *communicator, int *err); //////////////////////////////// // Update a partition after parts have been added. // Gives the implementation an opportunity to locally store info // about the partition so that queries on the partition can be // performed without synchronous communication. // TODO: Determine which values must be updated to allow other // TODO: interface functions to be performed without synchronous // TODO: communication. // Values that are precomputed by updatePartitionPar: // + The total number of parts in a partition. // + Mapping between part handles and processes. // COMMUNICATION: Collective. void prefix_updatePartitionPar(iMesh_Instance instance, prefix_PartitionHandle partition_handle, int *err); //////////////////////////////// // Given a mesh instance, return number of partition handles and // all partition handles associated with the mesh instance. // COMMUNICATION: None. void prefix_getNumPartitions(iMesh_Instance instance, /*out*/ int *num_partitions, int *err); // COMMUNICATION: None. void prefix_getPartitions(iMesh_Instance instance, /*inout*/ prefix_PartitionHandle **partition_handle, /*inout*/ int *partition_handle_allocated, /*out*/ int *partition_handle_size, int *err); //////////////////////////////// // Given a partition handle, return the total global number of parts // in the partition. // COMMUNICATION: None++. void prefix_getNumParts(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, int *num_global_part, int *err); //////////////////////////////// // Map from parts to processes: // Given a partition handle and a part handle, return the rank of the // process that owns the part. // The part_handle may be local or remote. // COMMUNICATION: None++. void prefix_getPartRank(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const prefix_PartHandle part_handle, /*out*/ int *rank, int *err); // COMMUNICATION: None++. void prefix_getPartRankArr(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const prefix_PartHandle *part_handle, /*in*/ const int part_handle_size, /*inout*/ int **rank, /*inout*/ int *rank_allocated, int *err); //////////////////////////////// // Map from processes to parts: // + Given a partition handle and a process rank, // return the number of parts owned by the process. // COMMUNICATION: None++. void prefix_getNumPartsOnRank(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int rank, /*out*/ int *num_parts, int *err); // COMMUNICATION: None++. void prefix_getNumPartsArrOnRank(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int *rank, /*in*/ const int rank_size, /*inout*/ int **num_parts, /*inout*/ int *num_parts_allocated, int *err); // + Given a partition handle and a process rank, // return the part handles owned by the process. // COMMUNICATION: None++. void prefix_getPartsOnRank(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int rank, /*inout*/ prefix_PartHandle **part_handles, /*inout*/ int *part_handles_allocated, /*out*/ int *part_handles_size, int *err); // COMMUNICATION: None++. void prefix_getPartsArrOnRank(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const int *rank, /*in*/ const int rank_size, /*inout*/ prefix_PartHandle **part_handles, /*inout*/ int *part_handles_allocated, /*out*/ int *part_handles_size, int *err); //////////////////////////////// // Provide global mesh information about a partition. // Note that these functions may require communication and, thus, // would have to be called by all processes in the partition handle. // Given a mesh instance and partition handle, return: // + Total number of entities with given type or topology in the partition. // NOTE: Tim asks: does it make sense to have an entity set handle parameter // NOTE: in these functions? Do entity sets given on each process have to be // NOTE: ones shared in parallel? // NOTE: KDD says: Yes, it makes sense; this function is analogous to // NOTE: iMesh_getNumOfType, // NOTE: which takes the root set or any other entity set. // NOTE: KDD adds: we agreed that entity sets could span across parts and // NOTE: processes. If entity set handles are, say, addresses, how does the // NOTE: implementation know that entity set handles on different processes // NOTE: actually are the same entity set? // NOTE: Carl, Lori and RPI said the implementations should be able to // NOTE: identify sets that are actually the same set spread across many // NOTE: parts. // TODO: WE'LL GET TIM'S OPINION BEFORE MAKING FINAL DECISION. // COMMUNICATION: Collective. void prefix_getNumOfTypePar(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, const iBase_EntitySetHandle entity_set_handle, const int entity_type, int *num_type, int *err); // COMMUNICATION: Collective. void prefix_getNumOfTopoPar(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, const iBase_EntitySetHandle entity_set_handle, const int entity_topology, int *num_topo, int *err); // + Total number of entity sets in the partition. // // COMMUNICATION: Collective. // void prefix_getNumEntSetsPar(iMesh_Instance instance, // const prefix_PartitionHandle partition_handle, // const iBase_EntitySetHandle entity_set_handle, // const int num_hops, int *num_sets, int *err); // // TODO: THIS FUNCTION WAS NOT ONE OF CARL'S SIXTEEN, BUT IT IS ANALOGOUS TO // TODO: HIS FUNCTIONS. DO WE NEED IT? // TODO: TIM SAYS "NO"; ENTITY SETS AREN'T IN PARTITIONS, SO THIS FUNCTION IS // TODO: POORLY DEFINED. // TODO: I'LL COMMENT IT OUT FOR NOW. BUT ARE THERE SCENARIOS WHERE AN // TODO: APPLICATION WOULD WANT TO KNOW THE NUMBER OF SETS WITH ENTITIES // TODO: IN A PART? /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ /* Part Functionality */ /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ //////////////////////////////// // Given a partition handle, create a part and add it to the // partition on the process invoking the creation. Return part handle. // COMMUNICATION: None. void prefix_createPart(iMesh_Instance instance, /*in*/ prefix_PartitionHandle partition_handle, /*out*/ prefix_PartHandle *part_handle, int *err); //////////////////////////////// // Given a partition handle and a part handle, remove the part // from the partition, destroy the part, and invalidate the part handle. // If part_handle is remote, an error is returned. // COMMUNICATION: None. void prefix_destroyPart(iMesh_Instance instance, /*in*/ prefix_PartitionHandle partition_handle, /*inout*/ prefix_PartHandle *part_handle, int *err); //////////////////////////////// // Identify parts that neighbor a given part. // + Given a partition handle, a part handle, and an entity type, // return the number of parts in the partition that neighbor the given part // (using the definition of "neighbor" above). // If part_handle is remote, an error is returned. // COMMUNICATION: None++. void prefix_getNumPartNbors(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const prefix_PartHandle part_handle, /*in*/ int entity_type, /*out*/ int *num_part_nbors, int *err); // COMMUNICATION: None++. void prefix_getNumPartNborsArr(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const prefix_PartHandle *part_handles, /*in*/ const int part_handles_size, /*in*/ int entity_type, /*inout*/ int **num_part_nbors, /*inout*/ int *num_part_nbors_allocated, int *err); // + Given a partition handle, a part handle, and an entity type, // return the part handles of // all parts in the partition that neighbor the given part // (using the definition of "neighbor" above). // If part_handle is remote, an error is returned. // COMMUNICATION: None++. void prefix_getPartNbors(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const prefix_PartHandle part_handle, /*in*/ int entity_type, /*inout*/ prefix_PartHandle **nbor_part_handles, /*inout*/ int *nbor_part_handles_allocated, /*out*/ int *nbor_part_handles_size, int *err); // COMMUNICATION: None++. void prefix_getPartNborsArr(iMesh_Instance instance, const prefix_PartitionHandle partition_handle, /*in*/ const prefix_PartHandle *part_handles, /*in*/ const int part_handles_size, /*in*/ int entity_type, /*inout*/ prefix_PartHandle **nbor_part_handles, /*inout*/ int *nbor_part_handles_allocated, /*out*/ int *nbor_part_handles_size, int *err); //////////////////////////////// // Provide part boundary info: // Note: Allow optional specification of desired entity types and // topologies; allow neighboring part handle = prefix_ALL_PARTS to // count/include all qualifying boundary entities of part. // + Given a partition handle, a part handle, entity type and topology, and a // neighboring part handle, return the number of qualifying entities on // the part boundary shared with the neighboring part. // If part_handle is remote, an error is returned. // COMMUNICATION: None++. void prefix_getNumPartBdryEnts(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const int entity_type, /*in*/ const int entity_topology, /*in*/ const prefix_PartHandle nbor_part_handle, /*out*/ int *num_nbor_entities, int *err); // + Given a partition handle, a part handle, entity type and topology, and a // neighboring part handle, return an array of entity handles for all // qualifying entities along the part boundary shared with the neighboring // part. // If part_handle is remote, an error is returned. // COMMUNICATION: None++. void prefix_getPartBdryEnts(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const int entity_type, /*in*/ const int entity_topology, /*in*/ const prefix_PartHandle nbor_part_handle, /*inout*/ iBase_EntityHandle **entity_handles, /*inout*/ int *entity_handles_allocated, /*out*/ int *entity_handles_size, int *err); // + Boundary iterators: Given a partition handle, a part handle, and a // neighboring part handle, return an iterator over all entities along // the part boundary shared with the neighboring part. // Functionality for getNext, reset, and end is // provided through the regular iMesh iterator functions. // If part_handle is remote, an error is returned. // COMMUNICATION: None++. void prefix_initPartBdryEntIter(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const int entity_type, /*in*/ const int entity_topology, /*in*/ const prefix_PartHandle nbor_part_handle, /*inout*/ iMesh_EntityIterator* entity_iterator, int* err); // COMMUNICATION: None++. void prefix_initPartBdryEntArrIter(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const int entity_type, /*in*/ const int entity_topology, /*in*/ const int array_size, /*in*/ const prefix_PartHandle nbor_part_handle, /*inout*/ iMesh_EntityIterator* entity_iterator, int* err); //////////////////////////////// // Add/remove on-process entities to/from on-process part: Given // a partition handle, an entity handle, and a part handle, add/remove the // entity to/from the part. // On-process add/remove can be accomplished by functions // that add/remove entities to/from EntitySets. // TODO: ONKAR SAYS #define CAN BE PROBLEMATIC FOR FUNCTION NAMES. // TODO: WE'LL WORK OUT THE RIGHT MECHANISM. // COMMUNICATION: None. #define prefix_addEntToPart iMesh_addEntToSet #define prefix_rmvEntFromPart iMesh_rmvEntFromSet #define prefix_addEntArrToPart iMesh_addEntArrToSet #define prefix_rmvEntArrFromPart iMesh_rmvEntArrFromSet //////////////////////////////// // Add entities to on-process and/or off-process parts: // Collective-communication version: // prefix_sendEntArrToPartsPar and prefix_recvEntArrToPartPar are // collective operations, to be called by all processes in the // partition's communicator. prefix_sendEntArrToPartsPar must be // called before prefix_recvEntArrToPartPar. The message_tag argument // pairs the send with the receive. // The receive blocks until all expected data is received. // The implementation has everything it needs to determine sends and // receives. If desired, it can send and post non-blocking receives. // Upon return, the application can do other stuff for latency hiding. // COMMUNICATION: Collective. void prefix_sendEntArrToPartsPar(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle *entity_handles, /*in*/ const int entity_handles_size, /*in*/ const prefix_PartHandle *target_part_handles, /*in*/ int command_code, // e.g., MIGRATE,COPY /*in*/ int update_ghost, // e.g., YES,NO /*in*/ int message_tag, int *err); // The implementation can post blocking receives or wait for non- // blocking receives previously posted. Targets are correct, because // there may be multiple parts on a processor and the application does // not need to know sources. The message_tag ties // a send and receive pair together. // COMMUNICATION: Collective. void prefix_recvEntArrToPartsPar(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*inout*/ iBase_EntityHandle **entity_handles, /*inout*/ int *entity_handles_size, /*out*/ int *entity_handles_allocated, /*inout*/ prefix_PartHandle *target_part_handles, /*out*/ int *target_part_handles_allocated, /*inout*/ int command_code, // e.g., MIGRATE,COPY /*in*/ int update_ghost, // e.g., YES,NO /*in*/ int message_tag, int *err); // TODO: Carl and Onkar are working on non-collective versions, including // TODO: mechanism for requesting entities, sending entities, and receiving // TODO: entities. // TODO: Tim is working on communication functions to send tag info. //////////////////////////////// // Provide entity information about a part. // Given an entity set handle // and a part handle, return the number of entities of given type/topo // that are in both the part and the entity set. // If part_handle is remote, an error is returned. // COMMUNICATION: None. void prefix_getNumOfType(iMesh_Instance instance, const prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int entity_type, int *num_type, int *err); // COMMUNICATION: None. void prefix_getNumOfTopo(iMesh_Instance instance, const prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int entity_topology, int *num_topo, int *err); // COMMUNICATION: None. // void prefix_getEntSets(iMesh_Instance instance, // const prefix_PartHandle part_handle, // const iBase_EntitySetHandle entity_set_handle, // const int num_hops, // iBase_EntitySetHandle** contained_set_handles, // int* contained_set_handles_allocated, // int* contained_set_handles_size, int *err); // // TODO: THE ABOVE FUNCTION WAS NOT ONE OF CARL'S SIXTEEN, BUT IT IS ANALOGOUS // TODO: TO HIS FUNCTIONS. DO WE NEED IT? // TODO: TIM SAYS "NO"; ENTITY SETS AREN'T IN PARTITIONS, SO THIS FUNCTION IS // TODO: POORLY DEFINED. // TODO: I'LL COMMENT IT OUT FOR NOW. BUT ARE THERE SCENARIOS WHERE // TODO: APPLICATIONS WOULD WANT TO LOOP OVER PARTS AND THEN, WITHIN EACH // TODO: PART, LOOP OVER THE ENTITY SETS IN THE PART? // Given an entity set handle // and a part handle, return vertex information for vertices // that are in both the part and the entity set. // If part_handle is remote, an error is returned. // COMMUNICATION: None. void prefix_getAllVtxCoords(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, double** coordinates, int* coordinates_allocated, int* coordinates_size, int** in_entity_set, int* in_entity_set_allocated, int* in_entity_set_size, int* storage_order, int *err); // COMMUNICATION: None. void prefix_getVtxCoordIndex(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int requested_entity_type, const int requested_entity_topology, const int entity_adjacency_type, int** offset, int* offset_allocated, int* offset_size, int** index, int* index_allocated, int* index_size, int** entity_topologies, int* entity_topologies_allocated, int* entity_topologies_size, int *err); // Given an entity set handle // and a part handle, return entity handles for entities // that are in both the part and the entity set. // If part_handle is remote, an error is returned. // COMMUNICATION: None. void prefix_getEntities(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int entity_type, const int entity_topology, iBase_EntityHandle** entity_handles, int* entity_handles_allocated, int* entity_handles_size, int *err); // COMMUNICATION: None. void prefix_getAdjEntities(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntityHandle entity_set_handle, const int entity_type_requestor, const int entity_topology_requestor, const int entity_type_requested, iBase_EntityHandle** adj_entity_handles, int* adj_entity_handles_allocated, int* adj_entity_handles_size, int** offset, int* offset_allocated, int* offset_size, int** in_entity_set, int* in_entity_set_allocated, int* in_entity_set_size, int *err); //////////////////////////////// // Provide entity and array iterators for entities within both a given part // and a given entity set. // Functionality for getNext, reset, and end is // provided through the regular iMesh iterator functions. // If part_handle is remote, an error is returned. // COMMUNICATION: None. void prefix_initEntIter(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int requested_entity_type, const int requested_entity_topology, iMesh_EntityIterator* entity_iterator, int *err); // COMMUNICATION: None. void prefix_initEntArrIter(iMesh_Instance instance, prefix_PartHandle part_handle, const iBase_EntitySetHandle entity_set_handle, const int requested_entity_type, const int requested_entity_topology, const int requested_array_size, iMesh_EntityArrIterator* entArr_iterator, int *err); /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ /* Entity Functionality */ /*------------------------------------------------------------------------*/ /*------------------------------------------------------------------------*/ //////////////////////////////// // Provide part information about an entity: Given an entity and a // partition handle, return the part handle of the part that owns the entity. // Return an error code if an entity is not in the partition. // TODO: SHOULD MORE PRECISELY DEFINE WHEN ERROR CODE IS RETURNED. // TODO: E.G., WHEN AN ENTITY IS NOT OWNED BY ANY PROCESSES IN THE // TODO: PARTITIONS COMMUNICATOR; WHEN AN ENTITY IS NOT "CONNECTED" TO // TODO: ANY ENTITIES USED TO GENERATE THE PARTITION. // COMMUNICATION: None++. void prefix_getEntOwnerPart(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ iBase_EntityHandle entity_handle, /*out*/ prefix_PartHandle *part_handle, int* err); // COMMUNICATION: None++. void prefix_getEntOwnerPartArr(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ iBase_EntityHandle *entity_handles, /*in*/ int entity_handles_size, /*inout*/ prefix_PartHandle **part_handles, /*inout*/ int *part_handles_allocated, int* err); //////////////////////////////// // Provide entity categorization within part. // + Given a partition handle, a part handle, and an entity handle, return a // flag indicating whether the entity is owned by the part. // If part_handle is remote, an error is returned. // COMMUNICATION: None. void prefix_isEntOwner(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const iBase_EntityHandle entity_handle, /*out*/ int* is_owner, int *err); // COMMUNICATION: None. void prefix_isEntOwnerArr(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const iBase_EntityHandle *entity_handles, /*in*/ const int entity_handles_size, /*inout*/ int** is_owner, /*inout*/ int* is_owner_allocated, int *err); // + Given a partition handle, a part handle, and an entity handle, return a // flag indicating whether the entity is strictly internal, on a // boundary, or a ghost. // If part_handle is remote, an error is returned. // COMMUNICATION: None. void prefix_isEntParStatus(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const iBase_EntityHandle entity_handle, /*out*/ int* par_status, // Values=INTERNAL,BOUNDARY,GHOST int *err); // COMMUNICATION: None. void prefix_isEntParStatusArr(iMesh_Instance instance, /*in*/ const prefix_PartHandle part_handle, /*in*/ const iBase_EntityHandle *entity_handles, /*in*/ const int entity_handles_size, /*inout*/ int** par_status, // Values=INTERNAL,BOUNDARY,GHOST /*inout*/ int* par_status_allocated, int *err); //////////////////////////////// // Provide information about copies of entities. // All these functions should work on the local process as well as // remote processes; entity handles returned are likely but not // necessarily remote. For example, for a valid entity handle, the // first function above should return an integer >= 1, and the third // function should return at least the owned entity handle. // The part_handle may be local or remote. // TODO: ONKAR SAYS THESE FUNCTIONS DO NOT REQUIRE COMMUNICATION, BUT // TODO: FOR MANY OF THEM, I DON'T UNDERSTAND HOW THEY DON'T. // + Given a partition handle, an entity handle and a part handle, return the // entity handle of the entity in the specified part. // Return an error if the entity does not exist in the specified part. // The part_handle may be local or remote. // TODO: LORI OBSERVES: I STRUGGLE WITH THE copyEntOnPart FUNCTIONS... I // TODO: DON'T FULLY UNDERSTAND WHAT THIS ADDS TO THE INTERFACE AND i THINK IT // TODO: COULD LEAD TO HEADACHES WHEN THINKING ABOUT CONSISTENCY OF // TODO: INFORMATION ACROSS PROCESSORS. // TODO: MANY OF THESE FUNCTIONS WILL CHANGE WHEN ONKAR AND CARL SUBMIT // TODO: THEIR NON-COLLECTIVE COMMUNICATION CALLS. DON'T WORRY ABOUT THESE // TODO: THIS WEEK. // COMMUNICATION: ???. Subject to Onkar & Carl's revisions. void prefix_getCopyEntOnPart(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*in*/ prefix_PartHandle* part_handle, /*out*/ iBase_EntityHandle* copy_entity_handle, int *err); // + Given a partition handle and an entity handle, return the number // of copies of the entity in the partition. // COMMUNICATION: None++. void prefix_getNumCopiesEnt(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*out*/ int *num_copies_ent, int *err); // + Given a partition handle and an entity handle, return the part handles // of copies of the entity in the partition. (Requested by Onkar.) // COMMUNICATION: None++. void prefix_getCopiesEntParts(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*inout*/ prefix_PartHandle **part_handles, /*inout*/ int *part_handles_allocated, /*out*/ int *part_handles_size, int *err); // + Given a partition handle and an entity handle, return (remote) entity // handles and part handles of all copies of the entity. // COMMUNICATION: ???. Subject to Onkar & Carl's revisions. void prefix_getCopiesEnt(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*inout*/ prefix_PartHandle **part_handles, /*inout*/ int *part_handles_allocated, /*out*/ int *part_handles_size, /*inout*/ iBase_EntityHandle **copies_entity_handles, /*inout*/ int *copies_entity_handles_allocated, int *err); // + Given a partition handle and an entity handle, return the entity // handle and part handle from the owner of the entity (e.g., the // entity handle of the copy that has right-to-modify). // COMMUNICATION: ???. Subject to Onkar & Carl's revisions. void prefix_getOwnedEnt(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*out*/ prefix_PartHandle *owner_part_handle, /*out*/ iBase_EntityHandle *owner_entity_handle, int *err); // + Given a partition handle, an entity handle, and a part handle, add/remove // copies of the entity to/from the specified part. // The part_handle may be local or remote. // TODO: TIM ASKS WHETHER THESE FUNCTIONS COMMUNICATE. I HAD ASSUMED // TODO: THEY DID, SINCE THEY TOOK PARTITION AND TARGET PART HANDLES // TODO: AS ARGUMENTS. BUT THEN THEY WOULD HAVE TO BE CALLED // TODO: SYNCHRONOUSLY. HOW WOULD THAT WORK? AND HOW WOULD THEY DIFFER // TODO: FROM prefix_sendEntArrToPartsPar/prefix_recvEntArrToPartsPar? // TODO: LORI OBSERVES: SEEMS AS THOUGH addCopyEnt/rmvCopyEnt COULD BE // TODO: COMBINED WITH THE OTHER ADD FUNCTIONS (THAT ARE LIKE ENTITYSET // TODO: FUNCTIONS) WITH AN APPROPRIATE ENUMERATOR (e.g. OWNED, COPY). // TODO: I THINK THAT ASSUMES THE FUNCTIONS WORK LOCALLY. // TODO: THERE IS TOO MUCH CONFUSION ON THESE FUNCTIONS // TODO: FOR ME TO MAKE CHANGES. WE'LL DISCUSS THEM IN A PHONE CONF. // TODO: WE NEED TO DETERMINE WHAT CAPABILITY WE NEED. // COMMUNICATION: ???. Subject to Onkar & Carl's revisions. void prefix_addCopyEnt(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*in*/ prefix_PartHandle target_part_handle, int *err); // COMMUNICATION: ???. Subject to Onkar & Carl's revisions. void prefix_rmvCopyEnt(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle entity_handle, /*in*/ prefix_PartHandle target_part_handle, int *err); /* ------------------------------------------------ Comments and resolved questions: - Applications will build partitions by (1) creating a partition handle on each process to be included in the partition; (2) adding parts to the partition handle within the process; (3) populating the parts with entities, and (4) calling prefix_UpdatePartitionPar to allow the implementation to compute global data for the partition. - All parts will have globally unique (across the partition's communicator) part handles. - For now, we will not include an iterator over local (to a process) parts within a partition. If it is needed, it can be added later. - We decided to add prefix_UpdatePartitionPar to precompute global and interpart data after a partition is modified. This update would allow many functions to operate without collective communication. - We will not provide capability to move entire parts to new processes; instead, the new process must create the part in its partition handle and then receive (perhaps in bulk) the entities to populate the part. In other words, parts can be added to only a local partition handle. - Currently, iMesh doesn't have the functionality to get entities or entity sets by type and tag in serial. Should it? Many people said it would be useful; others said it could be costly (in parallel) or numerically difficult (for floating point values). This issue is an iMesh issue, not a parallel interface issue, so for this document, the issue is resolved. The resolution: If iMesh adopts this capability, we will add it to the parallel interface. - Removed the following global queries; are there analogous serial iMesh functions? + Total number of entity sets with given type, tag and/or tag name in the partition; + Total entities with given type, tag and/or tag name in the partition; - We will not include functions that return all entities with given characteristics within a partition; the memory use of these functions can be large. Instead, we will return entity information with respect to parts and/or mesh instances. If the user wants such info, he should go through the mechanics of gathering it himself so that he is painfully aware of how much memory he is allocating. Removed the following global queries: + All tag names over the partition; + All entities in this partition having a given type, tag and/or tag name. + All entity sets in this partition having a given type, tag and/or tag name. - We will not include functions that return information about each part and/or process in a partition. Such functions limit memory scalability for large numbers of parts. If the user wants such info, he should go through the mechanics of gathering it himself so that he is painfully aware of how much memory he is allocating. Removed the following global queries: + The number of entities in each part of the partition; + The number of entity sets in each part of the partition; + The number of entities with given type, tag, and/or tag name in each part of the partition; + The number of entity sets with given type, tag, and/or tag name in each part of the partition; + All tag names in each part of the partition; - For functions that replace a set handle with a part handle, return all appropriate entities in a part, whether they are owned or are copies. The application can test for ownership if needed. - We decided that part assignments computed with respect to a set of entities induce part assignments to lower-dimensional entities in an implementation-dependent fashion. That is, if a partition is computed with respect to regions, queries about ownership of faces and vertices are valid. ------------------------------------------------ Discussed but unresolved questions: - We discussed and deferred a decision on adding functions that give hints to an implementation about which data mappings the application will use, allowing the implementation to pre-compute them if it chooses to. The example discussed was mapping between entities and parts, but other examples in iMesh may also exist. - We discussed and deferred a decision on adding an iterator over entities with given type/topology in a set or part. We have non-iterator functionality, but not an iterator. KDD: Is this true? What is iMesh_initEntIter (and its analogous KDD: prefix_initEntIter)? - We discussed and deferred a decision on storing in a partition information about which "objects" were used in computing the partition. These objects can be single entities or groups of entities. KDD: Perhaps this capability should be part of the load-balancing service. - We discussed and deferred a decision on designating a given partition as the "active" partition; i.e., the partition that is actually used in the distribution of mesh data in distributed memory. We were concerned that when multiple partitions were used, multiple copies of mesh entities would be needed to fully support multiple partitions at the same time. Designating one partition as "active" would store data with respect to only one partition. ------------------------------------------------ Not-yet-discussed, unresolved questions Entity questions: - From Carl: "getTag*Operate: Again, we haven't got this in serial. Does the existence of such operations imply that we expect to implement fields as tags? (Because that wasn't what I was assuming about field implementations at all, personally...) Note that I'm not opposed to this sort of global reduction operation, I just wonder whether it'll see use outside of field-like situations. If not, then it should be in parallel fields, not parallel mesh, and usage for fields-implemented-as-tags should be handled there." - Ghosting - File I/O + Do we need a mechanism to store a mapping of parts to processes in files? This capability could be used in static partitioning to compute on, say, 10 processes a partition with 10K parts that will be used on 5K processes. ------------------------------------------------ CVS File Information $RCSfile: DraftInterface.h,v $ $Author: kddevin $ $Date: 2008/02/22 18:55:26 $ $Revision: 1.14 $ ------------------------------------------------ */