itaps-parallel Notes from today's meeting

[Devine, Karen D]

The definitions are in the iMeshP.h header file; we can make them more
precise as needed.  Roughly,

- a "partition" is a high-level description of the distribution of mesh
entities; it includes a communicator, a list of local parts, a scheme for
mapping parts to processes, and methods for computing some global quantities
(e.g., global number of entities of given type).

- a part is a subset of entities; for all parts A and B in a given
partition, the intersection of the owned entities of A and B is NULL.

- "partitioning" is a verb; it is the act of creating an assignment of
entities to parts and can be done by, say, iZoltan or something equivalent.



On 4/19/10 9:24 AM, "seol at scorec.rpi.edu" <seol at scorec.rpi.edu> wrote:

>
>
> Do we have clear definition on the following terms?
>  - partitioning
>  - partition
>  - part
>
> Seegyoung
>
>> A couple comments on that
>> - If you constrain yourself to answering that question based on the
>> current iMeshP definition of "partition" you'll probably get a
>> different answer than if you don't
>> - Ideally, applications that don't need that shouldn't have to pay for
>> the ability to do it, either in performance (memory or speed) or in
>> complication of the API.
>>
>> I'll go back to what I've said before. I can't see an implementation
>> of this kind of thing (where the two partitions are not correlated)
>> where the second partition isn't effectively a copy of some of the
>> entities in the first partition that is then partitioned the way it's
>> needed.
>>
>>
>> mark
>>
>> On Apr 19, 2010, at 10:55 AM, Devine, Karen D wrote:
>>
>>>
>>> I think the issue is not, "Can contact be implemented with only one
>>> partition of a mesh?"  Rather, the issue is, "Some applications do
>>> contact
>>> with two separate partitions of a mesh.  Can iMeshP support that?"
>>>
>>> Karen
>>>
>>>
>>>
>>> On 4/19/10 8:48 AM, "Tim Tautges" <tautges at mcs.anl.gov> wrote:
>>>
>>>> The issue addressed in the work I'm citing had to do with load
>>>> balancing of
>>>> the surface vs. volume calculation, where
>>>> surface here means geometric surface, not interprocessor boundary.
>>>> In that
>>>> work, they needed to check for contact
>>>> frequently (maybe even every timestep?), but keep the volumetric
>>>> partition
>>>> also, so migration was out of the question.
>>>> That's why I call it different from migration before/after
>>>> adaptation.  I'm
>>>> sure there are multiple ways of implementing
>>>> it, though.
>>>>
>>>> - tim
>>>>
>>>>
>>>> Mark Shephard wrote:
>>>>> It is not hard at all. In fact it is quite straight forward and
>>>>> clean to
>>>>> add an implementation that already deals with fine level evolution
>>>>> of
>>>>> the simulation model.
>>>>>
>>>>> tautges at mcs.anl.gov wrote:
>>>>>> That sounds really hard, but ok.
>>>>>>
>>>>>> - tim
>>>>>>
>>>>>>
>>>>>> On Apr 19, 2010, at 8:55 AM, Mark Shephard
>>>>>> <shephard at scorec.rpi.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Tim,
>>>>>>>
>>>>>>> I am sure that we can effectively implement contact, including the
>>>>>>> evolution of contact and inclusion of mesh adaptation if desired
>>>>>>> as I
>>>>>>> have indicated (a single partition of the mesh). The issue of
>>>>>>> needing
>>>>>>> to migrate things between parts during adaptation has nothing to
>>>>>>> do
>>>>>>> with it. Some form of dynamic load balancing is to account for any
>>>>>>> change (contact evolution or mesh adaptation) if one wants to
>>>>>>> continue to scale. Again, that is independent of the other
>>>>>>> questions.
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>>
>>>>>>> tautges at mcs.anl.gov wrote:
>>>>>>>> I'm not sure this is true. If I understand your amr method
>>>>>>>> correctly, you only adapt the interior of meshes, migrating a
>>>>>>>> potion
>>>>>>>> of an interface to make that so.  Or, have you changed that?
>>>>>>>> Whereas with parallel contact (at least as implemented by sandia,
>>>>>>>> which I'm familiar with), the face partition is completely
>>>>>>>> unrelated
>>>>>>>> to the region partition.
>>>>>>>> - tim
>>>>>>>> On Apr 19, 2010, at 5:15 AM, Mark Shephard <shephard at scorec.rpi.edu
>>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>> Two quick comments/questions:
>>>>>>>>>
>>>>>>>>> Is there a conference call today, Monday? If yes, what time?
>>>>>>>>>
>>>>>>>>> The issues of doing a contact problem in parallel are really
>>>>>>>>> much
>>>>>>>>> the say as doing mesh adaptivity in parallel with a form of
>>>>>>>>> nonmanifold model. The only new item past that of a 2-manifold
>>>>>>>>> situation is keeping  track of the contact face interactions.
>>>>>>>>> There
>>>>>>>>> is absolutely no problem doing this within in the concept of a
>>>>>>>>> single partition of the mesh. On the other hand, considering
>>>>>>>>> how to
>>>>>>>>> track an evolving contact interface through the use of multiple
>>>>>>>>> partitions of the mesh would actually add substantial
>>>>>>>>> complexity to
>>>>>>>>> the mesh up-date and dynamic load balancing processes.
>>>>>>>>>
>>>>>>>>> Mark
>>>>>>>>>
>>>>>>>>> Carl Ollivier-Gooch wrote:
>>>>>>>>>> Tim Tautges wrote:
>>>>>>>>>>> I think Jason's argument is important here: the easy things
>>>>>>>>>>> should be easy, and more complicated things possible.  I think
>>>>>>>>>>> this is related to Seegyoung's point about having a single
>>>>>>>>>>> API.
>>>>>>>>>>> I think what this boils down to is an assertion that we should
>>>>>>>>>>> think primarily in terms of one iMesh Instance per process.
>>>>>>>>>>> For
>>>>>>>>>>> those applications wanting to use multiple instances, that
>>>>>>>>>>> implies communication between the instances at least
>>>>>>>>>>> conceptually, if not in the implementation also.  What this
>>>>>>>>>>> boils
>>>>>>>>>>> down to even further is that Part and iMesh instance are
>>>>>>>>>>> synonymous.  The issue of an entity assigned to exactly one
>>>>>>>>>>> Part
>>>>>>>>>>> morphs to one of representation vs. ownership: an entity can
>>>>>>>>>>> be
>>>>>>>>>>> represented in more than one instance (e.g. faces on part
>>>>>>>>>>> boundaries, in an element-based partition), but can be owned
>>>>>>>>>>> by
>>>>>>>>>>> only one of those instances.
>>>>>>>>>> Personally, I'm very much in favor of the part = instance
>>>>>>>>>> interpretation, for reasons already outlined.
>>>>>>>>>> In addition, I can easily see how we'll be able to implement
>>>>>>>>>> this,
>>>>>>>>>> whereas with multiple parts per instance, the easiest approach
>>>>>>>>>> would be for us, effectively, to have multiple of our current
>>>>>>>>>> instances inside a wrapper; yuck!
>>>>>>>>>> At a higher level, this also makes the whole parts-as-special-
>>>>>>>>>> sets
>>>>>>>>>> argument moot, saving us from possibly re-visiting that.  If
>>>>>>>>>> this
>>>>>>>>>> is the way we decide to go, we'll want to do an audit of iMeshP
>>>>>>>>>> functions, because part handles will now be synonymous with
>>>>>>>>>> iMesh
>>>>>>>>>> instance handles.  I suspect that a whole lot of functions
>>>>>>>>>> either
>>>>>>>>>> go away or get simpler.  This is not a bad thing.
>>>>>>>>>>> One of the reasons we initially thought in terms of multiple
>>>>>>>>>>> parts per process was to handle over-partitionings, where a
>>>>>>>>>>> process could be assigned multiple parts, e.g. for better load
>>>>>>>>>>> balancing, and flexibility in what's stored with a mesh or how
>>>>>>>>>>> it's stored.  I think that concept is an artifact of how a
>>>>>>>>>>> partitioning is stored with the mesh (or how the mesh is
>>>>>>>>>>> stored
>>>>>>>>>>> in files to reflect a partitioning).  From an application's
>>>>>>>>>>> point
>>>>>>>>>>> of view, in most cases it's still going to want to access the
>>>>>>>>>>> mesh from a single instance on a given process, no matter how
>>>>>>>>>>> that mesh was loaded initially.  That's also how the mesh
>>>>>>>>>>> looks
>>>>>>>>>>> from most applications' points of view.  The use case of
>>>>>>>>>>> multiple
>>>>>>>>>>> parts, with shared interface entities duplicated, is mostly
>>>>>>>>>>> handled by having separate entities in the database, with
>>>>>>>>>>> special
>>>>>>>>>>> relations between them implying that they're the same.
>>>>>>>>>> I can see ways around the over-partitioning problem, though.
>>>>>>>>>> Most
>>>>>>>>>> obviously, simply have multiple threads per processor, so
>>>>>>>>>> that the
>>>>>>>>>> number of processes still matches the number of parts.  Or an
>>>>>>>>>> app
>>>>>>>>>> or implementation (or even the interface) could specify how to
>>>>>>>>>> merge parts to get the right number at run time, as Tim hints
>>>>>>>>>> at.
>>>>>>>>>> Eventually, in the hyperparallel world that's coming, apps and
>>>>>>>>>> implementations may both have to be written to be multithreaded
>>>>>>>>>> shared-memory processes within a node (OpenMP?) with
>>>>>>>>>> communication
>>>>>>>>>> between nodes (MPI?).  This isn't in conflict with the notion
>>>>>>>>>> of
>>>>>>>>>> exactly one part (and iMesh instance) per process; it just
>>>>>>>>>> requires more careful programming of apps/services that use the
>>>>>>>>>> interface.
>>>>>>>>>>> One of the complications that arises from having one
>>>>>>>>>>> instance per
>>>>>>>>>>> process (and one part per instance) is how do you
>>>>>>>>>>> repartition, or
>>>>>>>>>>> partition in serial.  I think that concept is handled by the
>>>>>>>>>>> notion of initializing a given parallel mesh either a) from a
>>>>>>>>>>> collection of collections, or a partition of parts, stored
>>>>>>>>>>> with
>>>>>>>>>>> and loaded with the mesh, or b) communication from one
>>>>>>>>>>> instance
>>>>>>>>>>> to another, using parallel migration of entities.  Of
>>>>>>>>>>> course, a)
>>>>>>>>>>> is easily handled using sets, if you have them; if you don't,
>>>>>>>>>>> it's just as easily handled using multiple files.  It also
>>>>>>>>>>> naturally handles establishing a parallel mesh based on one of
>>>>>>>>>>> several "partitionings" (collection of collections) stored
>>>>>>>>>>> with a
>>>>>>>>>>> mesh, e.g. using the geometric volumes, material types, or
>>>>>>>>>>> true
>>>>>>>>>>> partitions (generated by a partitioner).
>>>>>>>>>> This is a problem we always had, though, didn't we?  I mean,
>>>>>>>>>> with
>>>>>>>>>> multiple instances per process, repartitioning to use a new
>>>>>>>>>> instance would require copying the whole mesh, including
>>>>>>>>>> adjacency, sets, tags, etc.  Do-able, but non-trivial,
>>>>>>>>>> especially
>>>>>>>>>> (say) tags with entity handle value.
>>>>>>>>>> I don't have any conceptual issues in principle with the
>>>>>>>>>> notion of
>>>>>>>>>> multiple partitionings of the same entities.  How an
>>>>>>>>>> implementation (or app) chooses to store the inactive
>>>>>>>>>> partition is
>>>>>>>>>> obviously going to be implementation-dependent (sets vs tags vs
>>>>>>>>>> classification vs ???).  This potentially makes it tricky to
>>>>>>>>>> define an API for switching between them.
>>>>>>>>>> Also, what about the different, more subtle case where we -
>>>>>>>>>> need-
>>>>>>>>>> two partitions active at once?  One use case is a contact
>>>>>>>>>> problem,
>>>>>>>>>> where the 3D mesh and 2D surface are partitioned separately
>>>>>>>>>> (I'm
>>>>>>>>>> assuming this is done, to get better load balance?).  If you're
>>>>>>>>>> also doing mesh adaptation as you go (likely), now those 2D
>>>>>>>>>> faces
>>>>>>>>>> on the interface have owners in the 3D partition (okay, some
>>>>>>>>>> implementation don't represent them explicitly, but it's safe
>>>>>>>>>> to
>>>>>>>>>> say that a face interior to a part is going to be owned by that
>>>>>>>>>> part...).  I see two alternatives.
>>>>>>>>>> 1. The easiest way to handle this in the current interfaces, as
>>>>>>>>>> far as parallel stuff goes, is to have two separate meshes (a
>>>>>>>>>> 3D
>>>>>>>>>> mesh and a 2D manifold mesh), with tags or iRel (or something
>>>>>>>>>> similar) keeping the respective entities associated with each
>>>>>>>>>> other.  This implies two iMesh instances per process, one
>>>>>>>>>> each for
>>>>>>>>>> the 3D and 2D manifold meshes, which isn't in conflict with
>>>>>>>>>> other
>>>>>>>>>> stuff in this thread.
>>>>>>>>>> 2. You could also have two partitions active within a single
>>>>>>>>>> instance. I'm not quite sure how this would look, even
>>>>>>>>>> conceptually.  You'd have the full 3D mesh for the (3D) part,
>>>>>>>>>> plus
>>>>>>>>>> a bunch of extra faces (and verts, at least) for the bits of
>>>>>>>>>> the
>>>>>>>>>> 2D manifold mesh that isn't already resident.  The problem with
>>>>>>>>>> this is that you've got to strictly segregate things so that
>>>>>>>>>> iteration over (say) faces gets you what you wanted.  Yes, I
>>>>>>>>>> know,
>>>>>>>>>> sets will do this, but it seems cumbersome...
>>>>>>>>>> I can't say I'm crazy about either of these alternatives, nor
>>>>>>>>>> am I
>>>>>>>>>> invested in either one.  Hopefully, someone can come up with
>>>>>>>>>> something better.  For what's it's worth, I think this
>>>>>>>>>> problem is
>>>>>>>>>> reasonably (completely?) independent of whether in "normal"
>>>>>>>>>> single
>>>>>>>>>> mesh contexts we require exactly one part and iMesh instance
>>>>>>>>>> per
>>>>>>>>>> process.
>>>>>>>>>> Having said that, I'll almost certainly be missing tomorrow's
>>>>>>>>>> telecon. I will, of course, be happy to kibitz afterwards by
>>>>>>>>>> email. :-)
>>>>>>>>>> Carl
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> ================================================================
>>>> "You will keep in perfect peace him whose mind is
>>>>   steadfast, because he trusts in you."               Isaiah 26:3
>>>>
>>>>              Tim Tautges            Argonne National Laboratory
>>>>          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>>>>          phone: (608) 263-8485      1500 Engineering Dr.
>>>>            fax: (608) 263-4499      Madison, WI 53706
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>
>