itaps-parallel Notes from today's meeting

[seol at scorec.rpi.edu]

> The iMeshP header file says "Partitions have communicators associated with
> them.  These communicators can be thought of as MPI communicators."  If we
> made iMesh instances == Parts, we would need to associate MPI
> communicators
> with iMesh instances, right?   Your application would not support the
> assumption of MPI_COMM_WORLD everywhere, right?

Some more additions: (please correct me if I am wrong)

  - By having mesh instance = part handle,
    1. we won't allow substituting part handle for entity set handle? My
guess is yes
    2. all iMeshP functions should be modified accordingly
      For example, iMeshP_createPart will be modified not to take both
iMesh_instance and iMeshP_PartHandle

  - The iMeshP doc says "A partition is a high-level description of the
distribution of mesh entities. A partition assigns entities to subsets
called parts" Based on the definition of partition which is a parallel
communicator abstraction, my interpretation is that the partition will
be something about "managing" distributed mesh entities rather than
describing the distribution of mesh entities.

  - MPI_COMM_WORLD is an MPI communicator, signifying that the command is
taking place within the full set of nodes. Considering multiple
partitions means multiple communicators,
   1. do we need to associate MPI communicators with iMesh instances?
   2. As we refine our iMeshP further, we shouldn't support the assumption
of MPI_COMM_WORLD everywhere.

Seegyoung


>
> The definitions are in the iMeshP.h header file; we can make them more
> precise as needed.  Roughly,
>
> - a "partition" is a high-level description of the distribution of mesh
> entities; it includes a communicator, a list of local parts, a scheme for
> mapping parts to processes, and methods for computing some global
> quantities
> (e.g., global number of entities of given type).
>
> - a part is a subset of entities; for all parts A and B in a given
> partition, the intersection of the owned entities of A and B is NULL.
>
> - "partitioning" is a verb; it is the act of creating an assignment of
> entities to parts and can be done by, say, iZoltan or something
> equivalent.
>
>
>
> On 4/19/10 9:24 AM, "seol at scorec.rpi.edu" <seol at scorec.rpi.edu> wrote:
>
>>
>>
>> Do we have clear definition on the following terms?
>>  - partitioning
>>  - partition
>>  - part
>>
>> Seegyoung
>>
>>> A couple comments on that
>>> - If you constrain yourself to answering that question based on the
>>> current iMeshP definition of "partition" you'll probably get a
>>> different answer than if you don't
>>> - Ideally, applications that don't need that shouldn't have to pay for
>>> the ability to do it, either in performance (memory or speed) or in
>>> complication of the API.
>>>
>>> I'll go back to what I've said before. I can't see an implementation
>>> of this kind of thing (where the two partitions are not correlated)
>>> where the second partition isn't effectively a copy of some of the
>>> entities in the first partition that is then partitioned the way it's
>>> needed.
>>>
>>>
>>> mark
>>>
>>> On Apr 19, 2010, at 10:55 AM, Devine, Karen D wrote:
>>>
>>>>
>>>> I think the issue is not, "Can contact be implemented with only one
>>>> partition of a mesh?"  Rather, the issue is, "Some applications do
>>>> contact
>>>> with two separate partitions of a mesh.  Can iMeshP support that?"
>>>>
>>>> Karen
>>>>
>>>>
>>>>
>>>> On 4/19/10 8:48 AM, "Tim Tautges" <tautges at mcs.anl.gov> wrote:
>>>>
>>>>> The issue addressed in the work I'm citing had to do with load
>>>>> balancing of
>>>>> the surface vs. volume calculation, where
>>>>> surface here means geometric surface, not interprocessor boundary.
>>>>> In that
>>>>> work, they needed to check for contact
>>>>> frequently (maybe even every timestep?), but keep the volumetric
>>>>> partition
>>>>> also, so migration was out of the question.
>>>>> That's why I call it different from migration before/after
>>>>> adaptation.  I'm
>>>>> sure there are multiple ways of implementing
>>>>> it, though.
>>>>>
>>>>> - tim
>>>>>
>>>>>
>>>>> Mark Shephard wrote:
>>>>>> It is not hard at all. In fact it is quite straight forward and
>>>>>> clean to
>>>>>> add an implementation that already deals with fine level evolution
>>>>>> of
>>>>>> the simulation model.
>>>>>>
>>>>>> tautges at mcs.anl.gov wrote:
>>>>>>> That sounds really hard, but ok.
>>>>>>>
>>>>>>> - tim
>>>>>>>
>>>>>>>
>>>>>>> On Apr 19, 2010, at 8:55 AM, Mark Shephard
>>>>>>> <shephard at scorec.rpi.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Tim,
>>>>>>>>
>>>>>>>> I am sure that we can effectively implement contact, including the
>>>>>>>> evolution of contact and inclusion of mesh adaptation if desired
>>>>>>>> as I
>>>>>>>> have indicated (a single partition of the mesh). The issue of
>>>>>>>> needing
>>>>>>>> to migrate things between parts during adaptation has nothing to
>>>>>>>> do
>>>>>>>> with it. Some form of dynamic load balancing is to account for any
>>>>>>>> change (contact evolution or mesh adaptation) if one wants to
>>>>>>>> continue to scale. Again, that is independent of the other
>>>>>>>> questions.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>>
>>>>>>>> tautges at mcs.anl.gov wrote:
>>>>>>>>> I'm not sure this is true. If I understand your amr method
>>>>>>>>> correctly, you only adapt the interior of meshes, migrating a
>>>>>>>>> potion
>>>>>>>>> of an interface to make that so.  Or, have you changed that?
>>>>>>>>> Whereas with parallel contact (at least as implemented by sandia,
>>>>>>>>> which I'm familiar with), the face partition is completely
>>>>>>>>> unrelated
>>>>>>>>> to the region partition.
>>>>>>>>> - tim
>>>>>>>>> On Apr 19, 2010, at 5:15 AM, Mark Shephard
>>>>>>>>> <shephard at scorec.rpi.edu
>>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>> Two quick comments/questions:
>>>>>>>>>>
>>>>>>>>>> Is there a conference call today, Monday? If yes, what time?
>>>>>>>>>>
>>>>>>>>>> The issues of doing a contact problem in parallel are really
>>>>>>>>>> much
>>>>>>>>>> the say as doing mesh adaptivity in parallel with a form of
>>>>>>>>>> nonmanifold model. The only new item past that of a 2-manifold
>>>>>>>>>> situation is keeping  track of the contact face interactions.
>>>>>>>>>> There
>>>>>>>>>> is absolutely no problem doing this within in the concept of a
>>>>>>>>>> single partition of the mesh. On the other hand, considering
>>>>>>>>>> how to
>>>>>>>>>> track an evolving contact interface through the use of multiple
>>>>>>>>>> partitions of the mesh would actually add substantial
>>>>>>>>>> complexity to
>>>>>>>>>> the mesh up-date and dynamic load balancing processes.
>>>>>>>>>>
>>>>>>>>>> Mark
>>>>>>>>>>
>>>>>>>>>> Carl Ollivier-Gooch wrote:
>>>>>>>>>>> Tim Tautges wrote:
>>>>>>>>>>>> I think Jason's argument is important here: the easy things
>>>>>>>>>>>> should be easy, and more complicated things possible.  I think
>>>>>>>>>>>> this is related to Seegyoung's point about having a single
>>>>>>>>>>>> API.
>>>>>>>>>>>> I think what this boils down to is an assertion that we should
>>>>>>>>>>>> think primarily in terms of one iMesh Instance per process.
>>>>>>>>>>>> For
>>>>>>>>>>>> those applications wanting to use multiple instances, that
>>>>>>>>>>>> implies communication between the instances at least
>>>>>>>>>>>> conceptually, if not in the implementation also.  What this
>>>>>>>>>>>> boils
>>>>>>>>>>>> down to even further is that Part and iMesh instance are
>>>>>>>>>>>> synonymous.  The issue of an entity assigned to exactly one
>>>>>>>>>>>> Part
>>>>>>>>>>>> morphs to one of representation vs. ownership: an entity can
>>>>>>>>>>>> be
>>>>>>>>>>>> represented in more than one instance (e.g. faces on part
>>>>>>>>>>>> boundaries, in an element-based partition), but can be owned
>>>>>>>>>>>> by
>>>>>>>>>>>> only one of those instances.
>>>>>>>>>>> Personally, I'm very much in favor of the part = instance
>>>>>>>>>>> interpretation, for reasons already outlined.
>>>>>>>>>>> In addition, I can easily see how we'll be able to implement
>>>>>>>>>>> this,
>>>>>>>>>>> whereas with multiple parts per instance, the easiest approach
>>>>>>>>>>> would be for us, effectively, to have multiple of our current
>>>>>>>>>>> instances inside a wrapper; yuck!
>>>>>>>>>>> At a higher level, this also makes the whole parts-as-special-
>>>>>>>>>>> sets
>>>>>>>>>>> argument moot, saving us from possibly re-visiting that.  If
>>>>>>>>>>> this
>>>>>>>>>>> is the way we decide to go, we'll want to do an audit of iMeshP
>>>>>>>>>>> functions, because part handles will now be synonymous with
>>>>>>>>>>> iMesh
>>>>>>>>>>> instance handles.  I suspect that a whole lot of functions
>>>>>>>>>>> either
>>>>>>>>>>> go away or get simpler.  This is not a bad thing.
>>>>>>>>>>>> One of the reasons we initially thought in terms of multiple
>>>>>>>>>>>> parts per process was to handle over-partitionings, where a
>>>>>>>>>>>> process could be assigned multiple parts, e.g. for better load
>>>>>>>>>>>> balancing, and flexibility in what's stored with a mesh or how
>>>>>>>>>>>> it's stored.  I think that concept is an artifact of how a
>>>>>>>>>>>> partitioning is stored with the mesh (or how the mesh is
>>>>>>>>>>>> stored
>>>>>>>>>>>> in files to reflect a partitioning).  From an application's
>>>>>>>>>>>> point
>>>>>>>>>>>> of view, in most cases it's still going to want to access the
>>>>>>>>>>>> mesh from a single instance on a given process, no matter how
>>>>>>>>>>>> that mesh was loaded initially.  That's also how the mesh
>>>>>>>>>>>> looks
>>>>>>>>>>>> from most applications' points of view.  The use case of
>>>>>>>>>>>> multiple
>>>>>>>>>>>> parts, with shared interface entities duplicated, is mostly
>>>>>>>>>>>> handled by having separate entities in the database, with
>>>>>>>>>>>> special
>>>>>>>>>>>> relations between them implying that they're the same.
>>>>>>>>>>> I can see ways around the over-partitioning problem, though.
>>>>>>>>>>> Most
>>>>>>>>>>> obviously, simply have multiple threads per processor, so
>>>>>>>>>>> that the
>>>>>>>>>>> number of processes still matches the number of parts.  Or an
>>>>>>>>>>> app
>>>>>>>>>>> or implementation (or even the interface) could specify how to
>>>>>>>>>>> merge parts to get the right number at run time, as Tim hints
>>>>>>>>>>> at.
>>>>>>>>>>> Eventually, in the hyperparallel world that's coming, apps and
>>>>>>>>>>> implementations may both have to be written to be multithreaded
>>>>>>>>>>> shared-memory processes within a node (OpenMP?) with
>>>>>>>>>>> communication
>>>>>>>>>>> between nodes (MPI?).  This isn't in conflict with the notion
>>>>>>>>>>> of
>>>>>>>>>>> exactly one part (and iMesh instance) per process; it just
>>>>>>>>>>> requires more careful programming of apps/services that use the
>>>>>>>>>>> interface.
>>>>>>>>>>>> One of the complications that arises from having one
>>>>>>>>>>>> instance per
>>>>>>>>>>>> process (and one part per instance) is how do you
>>>>>>>>>>>> repartition, or
>>>>>>>>>>>> partition in serial.  I think that concept is handled by the
>>>>>>>>>>>> notion of initializing a given parallel mesh either a) from a
>>>>>>>>>>>> collection of collections, or a partition of parts, stored
>>>>>>>>>>>> with
>>>>>>>>>>>> and loaded with the mesh, or b) communication from one
>>>>>>>>>>>> instance
>>>>>>>>>>>> to another, using parallel migration of entities.  Of
>>>>>>>>>>>> course, a)
>>>>>>>>>>>> is easily handled using sets, if you have them; if you don't,
>>>>>>>>>>>> it's just as easily handled using multiple files.  It also
>>>>>>>>>>>> naturally handles establishing a parallel mesh based on one of
>>>>>>>>>>>> several "partitionings" (collection of collections) stored
>>>>>>>>>>>> with a
>>>>>>>>>>>> mesh, e.g. using the geometric volumes, material types, or
>>>>>>>>>>>> true
>>>>>>>>>>>> partitions (generated by a partitioner).
>>>>>>>>>>> This is a problem we always had, though, didn't we?  I mean,
>>>>>>>>>>> with
>>>>>>>>>>> multiple instances per process, repartitioning to use a new
>>>>>>>>>>> instance would require copying the whole mesh, including
>>>>>>>>>>> adjacency, sets, tags, etc.  Do-able, but non-trivial,
>>>>>>>>>>> especially
>>>>>>>>>>> (say) tags with entity handle value.
>>>>>>>>>>> I don't have any conceptual issues in principle with the
>>>>>>>>>>> notion of
>>>>>>>>>>> multiple partitionings of the same entities.  How an
>>>>>>>>>>> implementation (or app) chooses to store the inactive
>>>>>>>>>>> partition is
>>>>>>>>>>> obviously going to be implementation-dependent (sets vs tags vs
>>>>>>>>>>> classification vs ???).  This potentially makes it tricky to
>>>>>>>>>>> define an API for switching between them.
>>>>>>>>>>> Also, what about the different, more subtle case where we -
>>>>>>>>>>> need-
>>>>>>>>>>> two partitions active at once?  One use case is a contact
>>>>>>>>>>> problem,
>>>>>>>>>>> where the 3D mesh and 2D surface are partitioned separately
>>>>>>>>>>> (I'm
>>>>>>>>>>> assuming this is done, to get better load balance?).  If you're
>>>>>>>>>>> also doing mesh adaptation as you go (likely), now those 2D
>>>>>>>>>>> faces
>>>>>>>>>>> on the interface have owners in the 3D partition (okay, some
>>>>>>>>>>> implementation don't represent them explicitly, but it's safe
>>>>>>>>>>> to
>>>>>>>>>>> say that a face interior to a part is going to be owned by that
>>>>>>>>>>> part...).  I see two alternatives.
>>>>>>>>>>> 1. The easiest way to handle this in the current interfaces, as
>>>>>>>>>>> far as parallel stuff goes, is to have two separate meshes (a
>>>>>>>>>>> 3D
>>>>>>>>>>> mesh and a 2D manifold mesh), with tags or iRel (or something
>>>>>>>>>>> similar) keeping the respective entities associated with each
>>>>>>>>>>> other.  This implies two iMesh instances per process, one
>>>>>>>>>>> each for
>>>>>>>>>>> the 3D and 2D manifold meshes, which isn't in conflict with
>>>>>>>>>>> other
>>>>>>>>>>> stuff in this thread.
>>>>>>>>>>> 2. You could also have two partitions active within a single
>>>>>>>>>>> instance. I'm not quite sure how this would look, even
>>>>>>>>>>> conceptually.  You'd have the full 3D mesh for the (3D) part,
>>>>>>>>>>> plus
>>>>>>>>>>> a bunch of extra faces (and verts, at least) for the bits of
>>>>>>>>>>> the
>>>>>>>>>>> 2D manifold mesh that isn't already resident.  The problem with
>>>>>>>>>>> this is that you've got to strictly segregate things so that
>>>>>>>>>>> iteration over (say) faces gets you what you wanted.  Yes, I
>>>>>>>>>>> know,
>>>>>>>>>>> sets will do this, but it seems cumbersome...
>>>>>>>>>>> I can't say I'm crazy about either of these alternatives, nor
>>>>>>>>>>> am I
>>>>>>>>>>> invested in either one.  Hopefully, someone can come up with
>>>>>>>>>>> something better.  For what's it's worth, I think this
>>>>>>>>>>> problem is
>>>>>>>>>>> reasonably (completely?) independent of whether in "normal"
>>>>>>>>>>> single
>>>>>>>>>>> mesh contexts we require exactly one part and iMesh instance
>>>>>>>>>>> per
>>>>>>>>>>> process.
>>>>>>>>>>> Having said that, I'll almost certainly be missing tomorrow's
>>>>>>>>>>> telecon. I will, of course, be happy to kibitz afterwards by
>>>>>>>>>>> email. :-)
>>>>>>>>>>> Carl
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> ================================================================
>>>>> "You will keep in perfect peace him whose mind is
>>>>>   steadfast, because he trusts in you."               Isaiah 26:3
>>>>>
>>>>>              Tim Tautges            Argonne National Laboratory
>>>>>          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>>>>>          phone: (608) 263-8485      1500 Engineering Dr.
>>>>>            fax: (608) 263-4499      Madison, WI 53706
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>
>
>
>