Reprentation of parts in iMeshP Karen Devine, SNL, 4/29/10 Thank you to everyone who participated in the iMeshP discussion on Wednesday. I learned a lot, and I appreciate the feedback. At the end of the discussion, I said I would draft what I felt was a "compromise" solution that would support a range of implementations. Fortunately, I have several implementations to consider as test cases -- Simmetrix's and GRUMMP's use of mesh instances as parts, MOAB's use of entity sets as parts, FMDB's use of its own part class. As with any "compromise," this one will not be perfect with respect to anyone's implementation or to the interface, but we want a compromise that is not too bad for anyone's implementation. I want to know the following: - Does this compromise work for your implementation? - If not, why not and what SPECIFICALLY is missing/present that prevents it from working? Throughout this document, I'll refer to our "current" interface (what is in our current iMeshP spec) and the "part-mesh" interface (the proposal to make parts equivalent to mesh instances). The goal is to support our current implementations (FMDB partition model, MOAB part-entitySet model) as well as enable efficient part-mesh-instance implementations in Simmetrix and GRUMMP. OK; here we go. ---------------------------------------------------------------------------- High-level overview: 1. Keep the typical use of one visible-to-the-application mesh instance per process. 2. Keep opaque part handles as in current iMeshP interface. 3. Keep most of the current iMeshP interface unchanged. 4. Keep partitions under mesh instances in the hierarchy of objects. 5. Add a new requirement that each mesh instance has at least one partition. 6. Add a new requirement that each entity must be in some part of some partition. ---------------------------------------------------------------------------- 1. Keep the typical use of one visible-to-the-application mesh instance per process. This assumption is not changed from our current interface. Looking back through my notes from iMeshP-design phone conferences, I see that the opposition to multiple meshes in a partition was strong from the outset. For example, on 11/15/07, I noted there was "major heartburn with multiple meshes in a partition." To see a detailed discussion of the issue, refer to our mailing list http://lists.mcs.anl.gov/pipermail/itaps-parallel/2007/date.html#245 The relevant emails have subject "Mesh Instances." In a nutshell, we decided that a "mesh instance" is in effect a "mesh database" that may contain one or more actual meshes. As such, it makes most sense to have one database per implementation per process. I apologize for not reviewing the mailing list and my notes before the bootcamp. [Off on a tangent: A comment was made in the thread that the name "mesh instance" is confusing, as it implies an instantiation of a single mesh, and I still agree with that comment, preferring "mesh database." But we are NOT arguing this point here, and I am NOT proposing to change the name, so do NOT reply about this short vent. There will be plenty more to reply about below. :)] Implications: - FMDB and MOAB already work this way, so they are unaffected by this point. - GRUMMP and Simmetrix are anticipating using mesh instances (or something very much like them) as parts. They will need some "uber" mesh instance that allows them to access their entire database; this "uber" mesh instance is the only one visible to the application. A very simplistic strategy would have the part-mesh-instances in an array indexed by a local part ID. The "uber" mesh instance would then be a pointer to the entire array. Operations performed over the entire "uber" mesh instance would have to loop over the part-mesh-instances. With this "uber" mesh instance, operations over all entities in a process are easier for the application than with the part-mesh interface (as the loop over parts is in the implementation, not the application). ---------------------------------------------------------------------------- 2. Keep opaque part handles as in current iMeshP interface. Opaque handles are used in ITAPS to let the implementations do what they want underneath the interface. To allow different implementations to represent parts differently, we will keep the opaque handles. This point is consistent with the rest of ITAPS and is necessary if we don't want to force a particular data type on parts. - FMDB and MOAB already work this way; MOAB returns entity set handles. - GRUMMP and Simmetrix could use part handles that are pointers to their part-mesh-instances, or, for the simplistic example above, part handles could be local indices into the mesh-instance array. ---------------------------------------------------------------------------- 3. Keep most of the current iMeshP interface unchanged. Since we are keeping opaque part handles, we'll need to keep most of the functions in the current iMeshP interface, rather than overload iMesh functions with mesh instances representing parts. So we won't be able to simplify the interface significantly. There will be a few modifications/additions as noted elsewhere in this document. ---------------------------------------------------------------------------- 4. Keep partitions under mesh instances in the hierarchy of objects. Since a mesh instance is actually a mesh database storing everything an implementation needs, partitions belong in the database and, thus, using the terms from my bootcamp viewgraphs, they are "under" mesh instances in the hierarchy of data types. A key point that I accepted in Nov 2007, and re-accepted after re-reading the email thread, is that a mesh instance is not a mesh; it is a database with the mesh instance acting as a "this" pointer to the C interface. ---------------------------------------------------------------------------- 5. Add a new requirement that each mesh instance has at least one partition. This requirement is needed to avoid the "limbo" state that causes efficiency problems in part-mesh-instance implementations. In truth, every mesh instance already has an implied partition. In serial, the root set is a partition with one part. In parallel, the implied partition has the same number of parts as processes (since parts do not extend outside a process). This requirement merely formalizes this implied partition. When a mesh instance is created, a partition is also created. The partition is populated with initially empty parts. The partition's handle can then be retrieved by iMeshP_getPartitions; the parts can be retrieved by iMeshP_getPartHandles. INTERFACE CHANGE: - New function iMeshP_newMeshAll accepts an MPI communicator and the number of desired parts per process in the default partition. (Note that new parts can be added later as well with iMeshP_addPart.) - For iMesh_newMesh, I see two possibilities. To emphasize the serial nature of iMesh, iMesh_newMesh could create a partition with communicator MPI_COMM_SELF and one part per process. Semantically, I think this possibility is most correct, as a serial iMesh does not have boundary entities, neighbor parts, etc. However, in applications currently using iMesh_newMesh, the assumed underlying partition is probably one part per process with communicator MPI_COMM_WORLD. Parallel applications should, more correctly, use iMeshP_newMeshAll. Implications: - We are pushing a new requirement into iMesh implementations. However, existing serial applications never have to know this default partition exists. Only applications calling iMeshP_getPartitions will see this partition. - iMesh_load populates the default partition. iMeshP_loadAll will populate a partition provided by a partition handle. - While it doesn't expose it, FMDB has a default partition to which entities are added. This requirement would expose their default so it can be returned by iMeshP_getPartitions. - While it doesn't expose it as a partition, MOAB has a default partition to which entities are added: the root sets on each process. This requirement would force MOAB to formally expose this implied partition so it can be returned by iMeshP_getPartitions. - For iMesh_newMesh, Simmetrix and GRUMMP could call iMeshP_newMeshAll with number of parts on process = 1 and communicator = MPI_COMM_SELF. - All implementations would need to implement the new function iMeshP_newMesh. ---------------------------------------------------------------------------- 6. Add a new requirement that each entity must be in some part of some partition. This requirement is needed to avoid the "limbo" state that causes efficiency problems in part-mesh-instance implementations. In the common use case of one part per process, all entities are already in an implied partition; they are in some root set on some process. This requirement merely formalizes this situation into a requirement. When entities are created, they are automatically added to some part of a partition. If they are created with iMesh_createEnt, the implementation can decide to which part they are added. For the common use case above, the decision is easy; the entities are added to the one part on the process. When there are multiple parts on a process, iMesh_createEnt can decide where to which part entities are added. INTERFACE CHANGE: To allow users more control when the number of parts or partitions on a process is greater than one, we will add iMeshP_createEnt, which takes a partition handle and part handle; the new entity is added to the part associated with the part handle. ---------------------------------------------------------------------------- What's missing: - At bootcamp we (re-)identified a gap in the iMeshP interface. iMeshP does not currently allow a user to specify whether a created entity is owned by a part, is a ghost, or is a copy. As it turns out, Carl raised this issue on 3/3/08, but we deferred deciding on it. Now is the time to decide! Is this capability needed, or can the implementation always figure it out? When would an application/service want to specify ownership? If it is needed, we can add a flag to iMeshP_createEnt.