itaps-parallel Notes from today's meeting

[Tim Tautges]

Precisely.

- tim

Devine, Karen D wrote:
> I think the issue is not, "Can contact be implemented with only one
> partition of a mesh?"  Rather, the issue is, "Some applications do contact
> with two separate partitions of a mesh.  Can iMeshP support that?"
> 
> Karen
> 
> 
> 
> On 4/19/10 8:48 AM, "Tim Tautges" <tautges at mcs.anl.gov> wrote:
> 
>> The issue addressed in the work I'm citing had to do with load balancing of
>> the surface vs. volume calculation, where
>> surface here means geometric surface, not interprocessor boundary.  In that
>> work, they needed to check for contact
>> frequently (maybe even every timestep?), but keep the volumetric partition
>> also, so migration was out of the question.
>> That's why I call it different from migration before/after adaptation.  I'm
>> sure there are multiple ways of implementing
>> it, though.
>>
>> - tim
>>
>>
>> Mark Shephard wrote:
>>> It is not hard at all. In fact it is quite straight forward and clean to
>>> add an implementation that already deals with fine level evolution of
>>> the simulation model.
>>>
>>> tautges at mcs.anl.gov wrote:
>>>> That sounds really hard, but ok.
>>>>
>>>> - tim
>>>>
>>>>
>>>> On Apr 19, 2010, at 8:55 AM, Mark Shephard <shephard at scorec.rpi.edu>
>>>> wrote:
>>>>
>>>>> Tim,
>>>>>
>>>>> I am sure that we can effectively implement contact, including the
>>>>> evolution of contact and inclusion of mesh adaptation if desired as I
>>>>> have indicated (a single partition of the mesh). The issue of needing
>>>>> to migrate things between parts during adaptation has nothing to do
>>>>> with it. Some form of dynamic load balancing is to account for any
>>>>> change (contact evolution or mesh adaptation) if one wants to
>>>>> continue to scale. Again, that is independent of the other questions.
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>> tautges at mcs.anl.gov wrote:
>>>>>> I'm not sure this is true. If I understand your amr method
>>>>>> correctly, you only adapt the interior of meshes, migrating a potion
>>>>>> of an interface to make that so.  Or, have you changed that?
>>>>>> Whereas with parallel contact (at least as implemented by sandia,
>>>>>> which I'm familiar with), the face partition is completely unrelated
>>>>>> to the region partition.
>>>>>> - tim
>>>>>> On Apr 19, 2010, at 5:15 AM, Mark Shephard <shephard at scorec.rpi.edu>
>>>>>> wrote:
>>>>>>> Two quick comments/questions:
>>>>>>>
>>>>>>> Is there a conference call today, Monday? If yes, what time?
>>>>>>>
>>>>>>> The issues of doing a contact problem in parallel are really much
>>>>>>> the say as doing mesh adaptivity in parallel with a form of
>>>>>>> nonmanifold model. The only new item past that of a 2-manifold
>>>>>>> situation is keeping  track of the contact face interactions. There
>>>>>>> is absolutely no problem doing this within in the concept of a
>>>>>>> single partition of the mesh. On the other hand, considering how to
>>>>>>> track an evolving contact interface through the use of multiple
>>>>>>> partitions of the mesh would actually add substantial complexity to
>>>>>>> the mesh up-date and dynamic load balancing processes.
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>> Carl Ollivier-Gooch wrote:
>>>>>>>> Tim Tautges wrote:
>>>>>>>>> I think Jason's argument is important here: the easy things
>>>>>>>>> should be easy, and more complicated things possible.  I think
>>>>>>>>> this is related to Seegyoung's point about having a single API.
>>>>>>>>> I think what this boils down to is an assertion that we should
>>>>>>>>> think primarily in terms of one iMesh Instance per process.  For
>>>>>>>>> those applications wanting to use multiple instances, that
>>>>>>>>> implies communication between the instances at least
>>>>>>>>> conceptually, if not in the implementation also.  What this boils
>>>>>>>>> down to even further is that Part and iMesh instance are
>>>>>>>>> synonymous.  The issue of an entity assigned to exactly one Part
>>>>>>>>> morphs to one of representation vs. ownership: an entity can be
>>>>>>>>> represented in more than one instance (e.g. faces on part
>>>>>>>>> boundaries, in an element-based partition), but can be owned by
>>>>>>>>> only one of those instances.
>>>>>>>> Personally, I'm very much in favor of the part = instance
>>>>>>>> interpretation, for reasons already outlined.
>>>>>>>> In addition, I can easily see how we'll be able to implement this,
>>>>>>>> whereas with multiple parts per instance, the easiest approach
>>>>>>>> would be for us, effectively, to have multiple of our current
>>>>>>>> instances inside a wrapper; yuck!
>>>>>>>> At a higher level, this also makes the whole parts-as-special-sets
>>>>>>>> argument moot, saving us from possibly re-visiting that.  If this
>>>>>>>> is the way we decide to go, we'll want to do an audit of iMeshP
>>>>>>>> functions, because part handles will now be synonymous with iMesh
>>>>>>>> instance handles.  I suspect that a whole lot of functions either
>>>>>>>> go away or get simpler.  This is not a bad thing.
>>>>>>>>> One of the reasons we initially thought in terms of multiple
>>>>>>>>> parts per process was to handle over-partitionings, where a
>>>>>>>>> process could be assigned multiple parts, e.g. for better load
>>>>>>>>> balancing, and flexibility in what's stored with a mesh or how
>>>>>>>>> it's stored.  I think that concept is an artifact of how a
>>>>>>>>> partitioning is stored with the mesh (or how the mesh is stored
>>>>>>>>> in files to reflect a partitioning).  From an application's point
>>>>>>>>> of view, in most cases it's still going to want to access the
>>>>>>>>> mesh from a single instance on a given process, no matter how
>>>>>>>>> that mesh was loaded initially.  That's also how the mesh looks
>>>>>>>>> from most applications' points of view.  The use case of multiple
>>>>>>>>> parts, with shared interface entities duplicated, is mostly
>>>>>>>>> handled by having separate entities in the database, with special
>>>>>>>>> relations between them implying that they're the same.
>>>>>>>> I can see ways around the over-partitioning problem, though.  Most
>>>>>>>> obviously, simply have multiple threads per processor, so that the
>>>>>>>> number of processes still matches the number of parts.  Or an app
>>>>>>>> or implementation (or even the interface) could specify how to
>>>>>>>> merge parts to get the right number at run time, as Tim hints at.
>>>>>>>> Eventually, in the hyperparallel world that's coming, apps and
>>>>>>>> implementations may both have to be written to be multithreaded
>>>>>>>> shared-memory processes within a node (OpenMP?) with communication
>>>>>>>> between nodes (MPI?).  This isn't in conflict with the notion of
>>>>>>>> exactly one part (and iMesh instance) per process; it just
>>>>>>>> requires more careful programming of apps/services that use the
>>>>>>>> interface.
>>>>>>>>> One of the complications that arises from having one instance per
>>>>>>>>> process (and one part per instance) is how do you repartition, or
>>>>>>>>> partition in serial.  I think that concept is handled by the
>>>>>>>>> notion of initializing a given parallel mesh either a) from a
>>>>>>>>> collection of collections, or a partition of parts, stored with
>>>>>>>>> and loaded with the mesh, or b) communication from one instance
>>>>>>>>> to another, using parallel migration of entities.  Of course, a)
>>>>>>>>> is easily handled using sets, if you have them; if you don't,
>>>>>>>>> it's just as easily handled using multiple files.  It also
>>>>>>>>> naturally handles establishing a parallel mesh based on one of
>>>>>>>>> several "partitionings" (collection of collections) stored with a
>>>>>>>>> mesh, e.g. using the geometric volumes, material types, or true
>>>>>>>>> partitions (generated by a partitioner).
>>>>>>>> This is a problem we always had, though, didn't we?  I mean, with
>>>>>>>> multiple instances per process, repartitioning to use a new
>>>>>>>> instance would require copying the whole mesh, including
>>>>>>>> adjacency, sets, tags, etc.  Do-able, but non-trivial, especially
>>>>>>>> (say) tags with entity handle value.
>>>>>>>> I don't have any conceptual issues in principle with the notion of
>>>>>>>> multiple partitionings of the same entities.  How an
>>>>>>>> implementation (or app) chooses to store the inactive partition is
>>>>>>>> obviously going to be implementation-dependent (sets vs tags vs
>>>>>>>> classification vs ???).  This potentially makes it tricky to
>>>>>>>> define an API for switching between them.
>>>>>>>> Also, what about the different, more subtle case where we -need-
>>>>>>>> two partitions active at once?  One use case is a contact problem,
>>>>>>>> where the 3D mesh and 2D surface are partitioned separately (I'm
>>>>>>>> assuming this is done, to get better load balance?).  If you're
>>>>>>>> also doing mesh adaptation as you go (likely), now those 2D faces
>>>>>>>> on the interface have owners in the 3D partition (okay, some
>>>>>>>> implementation don't represent them explicitly, but it's safe to
>>>>>>>> say that a face interior to a part is going to be owned by that
>>>>>>>> part...).  I see two alternatives.
>>>>>>>> 1. The easiest way to handle this in the current interfaces, as
>>>>>>>> far as parallel stuff goes, is to have two separate meshes (a 3D
>>>>>>>> mesh and a 2D manifold mesh), with tags or iRel (or something
>>>>>>>> similar) keeping the respective entities associated with each
>>>>>>>> other.  This implies two iMesh instances per process, one each for
>>>>>>>> the 3D and 2D manifold meshes, which isn't in conflict with other
>>>>>>>> stuff in this thread.
>>>>>>>> 2. You could also have two partitions active within a single
>>>>>>>> instance. I'm not quite sure how this would look, even
>>>>>>>> conceptually.  You'd have the full 3D mesh for the (3D) part, plus
>>>>>>>> a bunch of extra faces (and verts, at least) for the bits of the
>>>>>>>> 2D manifold mesh that isn't already resident.  The problem with
>>>>>>>> this is that you've got to strictly segregate things so that
>>>>>>>> iteration over (say) faces gets you what you wanted.  Yes, I know,
>>>>>>>> sets will do this, but it seems cumbersome...
>>>>>>>> I can't say I'm crazy about either of these alternatives, nor am I
>>>>>>>> invested in either one.  Hopefully, someone can come up with
>>>>>>>> something better.  For what's it's worth, I think this problem is
>>>>>>>> reasonably (completely?) independent of whether in "normal" single
>>>>>>>> mesh contexts we require exactly one part and iMesh instance per
>>>>>>>> process.
>>>>>>>> Having said that, I'll almost certainly be missing tomorrow's
>>>>>>>> telecon. I will, of course, be happy to kibitz afterwards by
>>>>>>>> email. :-)
>>>>>>>> Carl
>>>
>> --
>> ================================================================
>> "You will keep in perfect peace him whose mind is
>>    steadfast, because he trusts in you."               Isaiah 26:3
>>
>>               Tim Tautges            Argonne National Laboratory
>>           (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>>           phone: (608) 263-8485      1500 Engineering Dr.
>>             fax: (608) 263-4499      Madison, WI 53706
>>
>>
>>
> 
> 
> 

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706