itaps-parallel Notes from today's meeting

[seol at scorec.rpi.edu]

Do we have clear definition on the following terms?
 - partitioning
 - partition
 - part

Seegyoung

> A couple comments on that
> - If you constrain yourself to answering that question based on the
> current iMeshP definition of "partition" you'll probably get a
> different answer than if you don't
> - Ideally, applications that don't need that shouldn't have to pay for
> the ability to do it, either in performance (memory or speed) or in
> complication of the API.
>
> I'll go back to what I've said before. I can't see an implementation
> of this kind of thing (where the two partitions are not correlated)
> where the second partition isn't effectively a copy of some of the
> entities in the first partition that is then partitioned the way it's
> needed.
>
>
> mark
>
> On Apr 19, 2010, at 10:55 AM, Devine, Karen D wrote:
>
>>
>> I think the issue is not, "Can contact be implemented with only one
>> partition of a mesh?"  Rather, the issue is, "Some applications do
>> contact
>> with two separate partitions of a mesh.  Can iMeshP support that?"
>>
>> Karen
>>
>>
>>
>> On 4/19/10 8:48 AM, "Tim Tautges" <tautges at mcs.anl.gov> wrote:
>>
>>> The issue addressed in the work I'm citing had to do with load
>>> balancing of
>>> the surface vs. volume calculation, where
>>> surface here means geometric surface, not interprocessor boundary.
>>> In that
>>> work, they needed to check for contact
>>> frequently (maybe even every timestep?), but keep the volumetric
>>> partition
>>> also, so migration was out of the question.
>>> That's why I call it different from migration before/after
>>> adaptation.  I'm
>>> sure there are multiple ways of implementing
>>> it, though.
>>>
>>> - tim
>>>
>>>
>>> Mark Shephard wrote:
>>>> It is not hard at all. In fact it is quite straight forward and
>>>> clean to
>>>> add an implementation that already deals with fine level evolution
>>>> of
>>>> the simulation model.
>>>>
>>>> tautges at mcs.anl.gov wrote:
>>>>> That sounds really hard, but ok.
>>>>>
>>>>> - tim
>>>>>
>>>>>
>>>>> On Apr 19, 2010, at 8:55 AM, Mark Shephard
>>>>> <shephard at scorec.rpi.edu>
>>>>> wrote:
>>>>>
>>>>>> Tim,
>>>>>>
>>>>>> I am sure that we can effectively implement contact, including the
>>>>>> evolution of contact and inclusion of mesh adaptation if desired
>>>>>> as I
>>>>>> have indicated (a single partition of the mesh). The issue of
>>>>>> needing
>>>>>> to migrate things between parts during adaptation has nothing to
>>>>>> do
>>>>>> with it. Some form of dynamic load balancing is to account for any
>>>>>> change (contact evolution or mesh adaptation) if one wants to
>>>>>> continue to scale. Again, that is independent of the other
>>>>>> questions.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>> tautges at mcs.anl.gov wrote:
>>>>>>> I'm not sure this is true. If I understand your amr method
>>>>>>> correctly, you only adapt the interior of meshes, migrating a
>>>>>>> potion
>>>>>>> of an interface to make that so.  Or, have you changed that?
>>>>>>> Whereas with parallel contact (at least as implemented by sandia,
>>>>>>> which I'm familiar with), the face partition is completely
>>>>>>> unrelated
>>>>>>> to the region partition.
>>>>>>> - tim
>>>>>>> On Apr 19, 2010, at 5:15 AM, Mark Shephard <shephard at scorec.rpi.edu
>>>>>>> >
>>>>>>> wrote:
>>>>>>>> Two quick comments/questions:
>>>>>>>>
>>>>>>>> Is there a conference call today, Monday? If yes, what time?
>>>>>>>>
>>>>>>>> The issues of doing a contact problem in parallel are really
>>>>>>>> much
>>>>>>>> the say as doing mesh adaptivity in parallel with a form of
>>>>>>>> nonmanifold model. The only new item past that of a 2-manifold
>>>>>>>> situation is keeping  track of the contact face interactions.
>>>>>>>> There
>>>>>>>> is absolutely no problem doing this within in the concept of a
>>>>>>>> single partition of the mesh. On the other hand, considering
>>>>>>>> how to
>>>>>>>> track an evolving contact interface through the use of multiple
>>>>>>>> partitions of the mesh would actually add substantial
>>>>>>>> complexity to
>>>>>>>> the mesh up-date and dynamic load balancing processes.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>> Carl Ollivier-Gooch wrote:
>>>>>>>>> Tim Tautges wrote:
>>>>>>>>>> I think Jason's argument is important here: the easy things
>>>>>>>>>> should be easy, and more complicated things possible.  I think
>>>>>>>>>> this is related to Seegyoung's point about having a single
>>>>>>>>>> API.
>>>>>>>>>> I think what this boils down to is an assertion that we should
>>>>>>>>>> think primarily in terms of one iMesh Instance per process.
>>>>>>>>>> For
>>>>>>>>>> those applications wanting to use multiple instances, that
>>>>>>>>>> implies communication between the instances at least
>>>>>>>>>> conceptually, if not in the implementation also.  What this
>>>>>>>>>> boils
>>>>>>>>>> down to even further is that Part and iMesh instance are
>>>>>>>>>> synonymous.  The issue of an entity assigned to exactly one
>>>>>>>>>> Part
>>>>>>>>>> morphs to one of representation vs. ownership: an entity can
>>>>>>>>>> be
>>>>>>>>>> represented in more than one instance (e.g. faces on part
>>>>>>>>>> boundaries, in an element-based partition), but can be owned
>>>>>>>>>> by
>>>>>>>>>> only one of those instances.
>>>>>>>>> Personally, I'm very much in favor of the part = instance
>>>>>>>>> interpretation, for reasons already outlined.
>>>>>>>>> In addition, I can easily see how we'll be able to implement
>>>>>>>>> this,
>>>>>>>>> whereas with multiple parts per instance, the easiest approach
>>>>>>>>> would be for us, effectively, to have multiple of our current
>>>>>>>>> instances inside a wrapper; yuck!
>>>>>>>>> At a higher level, this also makes the whole parts-as-special-
>>>>>>>>> sets
>>>>>>>>> argument moot, saving us from possibly re-visiting that.  If
>>>>>>>>> this
>>>>>>>>> is the way we decide to go, we'll want to do an audit of iMeshP
>>>>>>>>> functions, because part handles will now be synonymous with
>>>>>>>>> iMesh
>>>>>>>>> instance handles.  I suspect that a whole lot of functions
>>>>>>>>> either
>>>>>>>>> go away or get simpler.  This is not a bad thing.
>>>>>>>>>> One of the reasons we initially thought in terms of multiple
>>>>>>>>>> parts per process was to handle over-partitionings, where a
>>>>>>>>>> process could be assigned multiple parts, e.g. for better load
>>>>>>>>>> balancing, and flexibility in what's stored with a mesh or how
>>>>>>>>>> it's stored.  I think that concept is an artifact of how a
>>>>>>>>>> partitioning is stored with the mesh (or how the mesh is
>>>>>>>>>> stored
>>>>>>>>>> in files to reflect a partitioning).  From an application's
>>>>>>>>>> point
>>>>>>>>>> of view, in most cases it's still going to want to access the
>>>>>>>>>> mesh from a single instance on a given process, no matter how
>>>>>>>>>> that mesh was loaded initially.  That's also how the mesh
>>>>>>>>>> looks
>>>>>>>>>> from most applications' points of view.  The use case of
>>>>>>>>>> multiple
>>>>>>>>>> parts, with shared interface entities duplicated, is mostly
>>>>>>>>>> handled by having separate entities in the database, with
>>>>>>>>>> special
>>>>>>>>>> relations between them implying that they're the same.
>>>>>>>>> I can see ways around the over-partitioning problem, though.
>>>>>>>>> Most
>>>>>>>>> obviously, simply have multiple threads per processor, so
>>>>>>>>> that the
>>>>>>>>> number of processes still matches the number of parts.  Or an
>>>>>>>>> app
>>>>>>>>> or implementation (or even the interface) could specify how to
>>>>>>>>> merge parts to get the right number at run time, as Tim hints
>>>>>>>>> at.
>>>>>>>>> Eventually, in the hyperparallel world that's coming, apps and
>>>>>>>>> implementations may both have to be written to be multithreaded
>>>>>>>>> shared-memory processes within a node (OpenMP?) with
>>>>>>>>> communication
>>>>>>>>> between nodes (MPI?).  This isn't in conflict with the notion
>>>>>>>>> of
>>>>>>>>> exactly one part (and iMesh instance) per process; it just
>>>>>>>>> requires more careful programming of apps/services that use the
>>>>>>>>> interface.
>>>>>>>>>> One of the complications that arises from having one
>>>>>>>>>> instance per
>>>>>>>>>> process (and one part per instance) is how do you
>>>>>>>>>> repartition, or
>>>>>>>>>> partition in serial.  I think that concept is handled by the
>>>>>>>>>> notion of initializing a given parallel mesh either a) from a
>>>>>>>>>> collection of collections, or a partition of parts, stored
>>>>>>>>>> with
>>>>>>>>>> and loaded with the mesh, or b) communication from one
>>>>>>>>>> instance
>>>>>>>>>> to another, using parallel migration of entities.  Of
>>>>>>>>>> course, a)
>>>>>>>>>> is easily handled using sets, if you have them; if you don't,
>>>>>>>>>> it's just as easily handled using multiple files.  It also
>>>>>>>>>> naturally handles establishing a parallel mesh based on one of
>>>>>>>>>> several "partitionings" (collection of collections) stored
>>>>>>>>>> with a
>>>>>>>>>> mesh, e.g. using the geometric volumes, material types, or
>>>>>>>>>> true
>>>>>>>>>> partitions (generated by a partitioner).
>>>>>>>>> This is a problem we always had, though, didn't we?  I mean,
>>>>>>>>> with
>>>>>>>>> multiple instances per process, repartitioning to use a new
>>>>>>>>> instance would require copying the whole mesh, including
>>>>>>>>> adjacency, sets, tags, etc.  Do-able, but non-trivial,
>>>>>>>>> especially
>>>>>>>>> (say) tags with entity handle value.
>>>>>>>>> I don't have any conceptual issues in principle with the
>>>>>>>>> notion of
>>>>>>>>> multiple partitionings of the same entities.  How an
>>>>>>>>> implementation (or app) chooses to store the inactive
>>>>>>>>> partition is
>>>>>>>>> obviously going to be implementation-dependent (sets vs tags vs
>>>>>>>>> classification vs ???).  This potentially makes it tricky to
>>>>>>>>> define an API for switching between them.
>>>>>>>>> Also, what about the different, more subtle case where we -
>>>>>>>>> need-
>>>>>>>>> two partitions active at once?  One use case is a contact
>>>>>>>>> problem,
>>>>>>>>> where the 3D mesh and 2D surface are partitioned separately
>>>>>>>>> (I'm
>>>>>>>>> assuming this is done, to get better load balance?).  If you're
>>>>>>>>> also doing mesh adaptation as you go (likely), now those 2D
>>>>>>>>> faces
>>>>>>>>> on the interface have owners in the 3D partition (okay, some
>>>>>>>>> implementation don't represent them explicitly, but it's safe
>>>>>>>>> to
>>>>>>>>> say that a face interior to a part is going to be owned by that
>>>>>>>>> part...).  I see two alternatives.
>>>>>>>>> 1. The easiest way to handle this in the current interfaces, as
>>>>>>>>> far as parallel stuff goes, is to have two separate meshes (a
>>>>>>>>> 3D
>>>>>>>>> mesh and a 2D manifold mesh), with tags or iRel (or something
>>>>>>>>> similar) keeping the respective entities associated with each
>>>>>>>>> other.  This implies two iMesh instances per process, one
>>>>>>>>> each for
>>>>>>>>> the 3D and 2D manifold meshes, which isn't in conflict with
>>>>>>>>> other
>>>>>>>>> stuff in this thread.
>>>>>>>>> 2. You could also have two partitions active within a single
>>>>>>>>> instance. I'm not quite sure how this would look, even
>>>>>>>>> conceptually.  You'd have the full 3D mesh for the (3D) part,
>>>>>>>>> plus
>>>>>>>>> a bunch of extra faces (and verts, at least) for the bits of
>>>>>>>>> the
>>>>>>>>> 2D manifold mesh that isn't already resident.  The problem with
>>>>>>>>> this is that you've got to strictly segregate things so that
>>>>>>>>> iteration over (say) faces gets you what you wanted.  Yes, I
>>>>>>>>> know,
>>>>>>>>> sets will do this, but it seems cumbersome...
>>>>>>>>> I can't say I'm crazy about either of these alternatives, nor
>>>>>>>>> am I
>>>>>>>>> invested in either one.  Hopefully, someone can come up with
>>>>>>>>> something better.  For what's it's worth, I think this
>>>>>>>>> problem is
>>>>>>>>> reasonably (completely?) independent of whether in "normal"
>>>>>>>>> single
>>>>>>>>> mesh contexts we require exactly one part and iMesh instance
>>>>>>>>> per
>>>>>>>>> process.
>>>>>>>>> Having said that, I'll almost certainly be missing tomorrow's
>>>>>>>>> telecon. I will, of course, be happy to kibitz afterwards by
>>>>>>>>> email. :-)
>>>>>>>>> Carl
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> ================================================================
>>> "You will keep in perfect peace him whose mind is
>>>   steadfast, because he trusts in you."               Isaiah 26:3
>>>
>>>              Tim Tautges            Argonne National Laboratory
>>>          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>>>          phone: (608) 263-8485      1500 Engineering Dr.
>>>            fax: (608) 263-4499      Madison, WI 53706
>>>
>>>
>>>
>>
>>
>
>