itaps-parallel Notes from today's meeting

[Tim Tautges]

Sorry for the radio silence.  Some random thoughts:

There seems to be dual meanings to Partition/Part/entities in a Part: one meaning is the collection of things sent to a 
partitioner like Zoltan and grouped into Parts.  The other implies ownership of each entity, including other-dimensional 
entities adjacent to the entities sent to Zoltan (I say other here instead of lower to cover vertex-based partitions as 
well as region-based partitions).

I think Jason's argument is important here: the easy things should be easy, and more complicated things possible.  I 
think this is related to Seegyoung's point about having a single API.  I think what this boils down to is an assertion 
that we should think primarily in terms of one iMesh Instance per process.  For those applications wanting to use 
multiple instances, that implies communication between the instances at least conceptually, if not in the implementation 
also.  What this boils down to even further is that Part and iMesh instance are synonymous.  The issue of an entity 
assigned to exactly one Part morphs to one of representation vs. ownership: an entity can be represented in more than 
one instance (e.g. faces on part boundaries, in an element-based partition), but can be owned by only one of those 
instances.

One of the reasons we initially thought in terms of multiple parts per process was to handle over-partitionings, where a 
process could be assigned multiple parts, e.g. for better load balancing, and flexibility in what's stored with a mesh 
or how it's stored.  I think that concept is an artifact of how a partitioning is stored with the mesh (or how the mesh 
is stored in files to reflect a partitioning).  From an application's point of view, in most cases it's still going to 
want to access the mesh from a single instance on a given process, no matter how that mesh was loaded initially.  That's 
also how the mesh looks from most applications' points of view.  The use case of multiple parts, with shared interface 
entities duplicated, is mostly handled by having separate entities in the database, with special relations between them 
implying that they're the same.

One of the complications that arises from having one instance per process (and one part per instance) is how do you 
repartition, or partition in serial.  I think that concept is handled by the notion of initializing a given parallel 
mesh either a) from a collection of collections, or a partition of parts, stored with and loaded with the mesh, or b) 
communication from one instance to another, using parallel migration of entities.  Of course, a) is easily handled using 
sets, if you have them; if you don't, it's just as easily handled using multiple files.  It also naturally handles 
establishing a parallel mesh based on one of several "partitionings" (collection of collections) stored with a mesh, 
e.g. using the geometric volumes, material types, or true partitions (generated by a partitioner).

It's almost sounding to me like I'm agreeing mostly with Mark B's view of things, with the added concept of loading a 
parallel mesh based on some collection structure.  The next question becomes whether we can standardize the method for 
specifying that collectionn structure from applications.  I can describe how MOAB does that now, that works for the 
three types of partitions mentioned above, if anybody's interested.

- tim


Devine, Karen D wrote:
> Notes from 4/12/10 phone conference attended by
> 
>   RPI:  Mark S., S. Yung (spelling?), Ting
>   Uwisc:  Tim T, Jason
>   UBC:  Carl
>   SNL:  Vitus, Karen
>   LLNL:  Mark M.
>   Simmetrix:  Mark B., Saurabh Tendulkar
> 
> Agreed:  We will meet again on 4/19 at 8:30am PST.
> 
> Action Items:  
> 
> 1.  Before our next meeting, write your view of the first big question below
> and email it to the list.  Include pros, cons, how apps will use it, how it
> will affect implementations, etc.  Mail it to
> itaps-parallel at lists.mcs.anl.gov.
> 
> 2.  Read other peoples' write-ups.  Definitely reply for clarification of
> details, but let's save the discussion for the phone call where it is easier
> to follow.
> 
> My notes from the meeting:
> 
> Agreed:  From the iMeshP perspective, an application will typically have,
> for a given mesh, one iMesh_instance per process, containing all entities
> stored on that process.  The iMesh_instance may have a partition associated
> with it, with zero, one or multiple parts per process.  The underlying
> implementation may implement this design in any way desired.
> 
> If I understand correctly, Simmetrix intends to implement multiple parts per
> process by having, internally, multiple "meshes," each representing a part.
> The iMesh functions would return results from the union of these meshes; the
> iMeshP part-specific functions would return results from individual meshes.
> (Does that sound correct, Mark B. and Saurabh?)
> 
> The first big question:  Can an entity exist in an iMesh_instance that has a
> partition without being owned by any part?
> Discussion:  There are several points in iMeshP operations where an entity
> may not (yet) have a part assignment.  For example, entities are created
> with iMesh_createEnt; they can then be added to a part with
> iMesh_addEntToSet (overloaded with a part handle).  But in-between the two
> function calls, the entity exists without being in a part.  Some think this
> state is acceptable, with the understanding that parallel operations are not
> possible on entities that are not in parts.  But this state causes problems
> for some implementations, which assume an entity is always in some part
> (with a default part assignment set during entity creation).  If we require
> that an entity always be assigned to some part, we'll likely need additional
> interface functions, such as iMeshP_createEnt that creates an entity and
> assigns it to a part.  Even if we don't have the requirement, we could
> consider adding such functions as a convenience.
> 
> The second big question:  How should we enable multiple partitions of a
> single mesh in iMeshP?
> 
> The third question:  Should iMeshP_destroyPart should return an error if the
> part to be destroyed is not empty?
> The answer to this question depends, to some extent, on our answer to the
> first big question above.
> Cons:  Requires users to explicitly remove entities from parts before
>        deleting the parts.  This operation requires three function
>        calls:  iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with
>        part handle), iMeshP_destroyPart.
> Pros:  Prevents users from shooting themselves in the foot.
>        Users would possibly call iMeshP_getEntities and
>        iMesh_rmvEntFromSet anyway to do migration to new part.
> 
> 
> 

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706