itaps-parallel Notes from today's meeting

[Devine, Karen D]

I think the issue is not, "Can contact be implemented with only one
partition of a mesh?"  Rather, the issue is, "Some applications do contact
with two separate partitions of a mesh.  Can iMeshP support that?"

Karen



On 4/19/10 8:48 AM, "Tim Tautges" <tautges at mcs.anl.gov> wrote:

> The issue addressed in the work I'm citing had to do with load balancing of
> the surface vs. volume calculation, where
> surface here means geometric surface, not interprocessor boundary.  In that
> work, they needed to check for contact
> frequently (maybe even every timestep?), but keep the volumetric partition
> also, so migration was out of the question.
> That's why I call it different from migration before/after adaptation.  I'm
> sure there are multiple ways of implementing
> it, though.
> 
> - tim
> 
> 
> Mark Shephard wrote:
>> It is not hard at all. In fact it is quite straight forward and clean to
>> add an implementation that already deals with fine level evolution of
>> the simulation model.
>> 
>> tautges at mcs.anl.gov wrote:
>>> That sounds really hard, but ok.
>>> 
>>> - tim
>>> 
>>> 
>>> On Apr 19, 2010, at 8:55 AM, Mark Shephard <shephard at scorec.rpi.edu>
>>> wrote:
>>> 
>>>> Tim,
>>>> 
>>>> I am sure that we can effectively implement contact, including the
>>>> evolution of contact and inclusion of mesh adaptation if desired as I
>>>> have indicated (a single partition of the mesh). The issue of needing
>>>> to migrate things between parts during adaptation has nothing to do
>>>> with it. Some form of dynamic load balancing is to account for any
>>>> change (contact evolution or mesh adaptation) if one wants to
>>>> continue to scale. Again, that is independent of the other questions.
>>>> 
>>>> Mark
>>>> 
>>>> 
>>>> tautges at mcs.anl.gov wrote:
>>>>> I'm not sure this is true. If I understand your amr method
>>>>> correctly, you only adapt the interior of meshes, migrating a potion
>>>>> of an interface to make that so.  Or, have you changed that?
>>>>> Whereas with parallel contact (at least as implemented by sandia,
>>>>> which I'm familiar with), the face partition is completely unrelated
>>>>> to the region partition.
>>>>> - tim
>>>>> On Apr 19, 2010, at 5:15 AM, Mark Shephard <shephard at scorec.rpi.edu>
>>>>> wrote:
>>>>>> Two quick comments/questions:
>>>>>> 
>>>>>> Is there a conference call today, Monday? If yes, what time?
>>>>>> 
>>>>>> The issues of doing a contact problem in parallel are really much
>>>>>> the say as doing mesh adaptivity in parallel with a form of
>>>>>> nonmanifold model. The only new item past that of a 2-manifold
>>>>>> situation is keeping  track of the contact face interactions. There
>>>>>> is absolutely no problem doing this within in the concept of a
>>>>>> single partition of the mesh. On the other hand, considering how to
>>>>>> track an evolving contact interface through the use of multiple
>>>>>> partitions of the mesh would actually add substantial complexity to
>>>>>> the mesh up-date and dynamic load balancing processes.
>>>>>> 
>>>>>> Mark
>>>>>> 
>>>>>> Carl Ollivier-Gooch wrote:
>>>>>>> Tim Tautges wrote:
>>>>>>>> I think Jason's argument is important here: the easy things
>>>>>>>> should be easy, and more complicated things possible.  I think
>>>>>>>> this is related to Seegyoung's point about having a single API.
>>>>>>>> I think what this boils down to is an assertion that we should
>>>>>>>> think primarily in terms of one iMesh Instance per process.  For
>>>>>>>> those applications wanting to use multiple instances, that
>>>>>>>> implies communication between the instances at least
>>>>>>>> conceptually, if not in the implementation also.  What this boils
>>>>>>>> down to even further is that Part and iMesh instance are
>>>>>>>> synonymous.  The issue of an entity assigned to exactly one Part
>>>>>>>> morphs to one of representation vs. ownership: an entity can be
>>>>>>>> represented in more than one instance (e.g. faces on part
>>>>>>>> boundaries, in an element-based partition), but can be owned by
>>>>>>>> only one of those instances.
>>>>>>> Personally, I'm very much in favor of the part = instance
>>>>>>> interpretation, for reasons already outlined.
>>>>>>> In addition, I can easily see how we'll be able to implement this,
>>>>>>> whereas with multiple parts per instance, the easiest approach
>>>>>>> would be for us, effectively, to have multiple of our current
>>>>>>> instances inside a wrapper; yuck!
>>>>>>> At a higher level, this also makes the whole parts-as-special-sets
>>>>>>> argument moot, saving us from possibly re-visiting that.  If this
>>>>>>> is the way we decide to go, we'll want to do an audit of iMeshP
>>>>>>> functions, because part handles will now be synonymous with iMesh
>>>>>>> instance handles.  I suspect that a whole lot of functions either
>>>>>>> go away or get simpler.  This is not a bad thing.
>>>>>>>> One of the reasons we initially thought in terms of multiple
>>>>>>>> parts per process was to handle over-partitionings, where a
>>>>>>>> process could be assigned multiple parts, e.g. for better load
>>>>>>>> balancing, and flexibility in what's stored with a mesh or how
>>>>>>>> it's stored.  I think that concept is an artifact of how a
>>>>>>>> partitioning is stored with the mesh (or how the mesh is stored
>>>>>>>> in files to reflect a partitioning).  From an application's point
>>>>>>>> of view, in most cases it's still going to want to access the
>>>>>>>> mesh from a single instance on a given process, no matter how
>>>>>>>> that mesh was loaded initially.  That's also how the mesh looks
>>>>>>>> from most applications' points of view.  The use case of multiple
>>>>>>>> parts, with shared interface entities duplicated, is mostly
>>>>>>>> handled by having separate entities in the database, with special
>>>>>>>> relations between them implying that they're the same.
>>>>>>> I can see ways around the over-partitioning problem, though.  Most
>>>>>>> obviously, simply have multiple threads per processor, so that the
>>>>>>> number of processes still matches the number of parts.  Or an app
>>>>>>> or implementation (or even the interface) could specify how to
>>>>>>> merge parts to get the right number at run time, as Tim hints at.
>>>>>>> Eventually, in the hyperparallel world that's coming, apps and
>>>>>>> implementations may both have to be written to be multithreaded
>>>>>>> shared-memory processes within a node (OpenMP?) with communication
>>>>>>> between nodes (MPI?).  This isn't in conflict with the notion of
>>>>>>> exactly one part (and iMesh instance) per process; it just
>>>>>>> requires more careful programming of apps/services that use the
>>>>>>> interface.
>>>>>>>> One of the complications that arises from having one instance per
>>>>>>>> process (and one part per instance) is how do you repartition, or
>>>>>>>> partition in serial.  I think that concept is handled by the
>>>>>>>> notion of initializing a given parallel mesh either a) from a
>>>>>>>> collection of collections, or a partition of parts, stored with
>>>>>>>> and loaded with the mesh, or b) communication from one instance
>>>>>>>> to another, using parallel migration of entities.  Of course, a)
>>>>>>>> is easily handled using sets, if you have them; if you don't,
>>>>>>>> it's just as easily handled using multiple files.  It also
>>>>>>>> naturally handles establishing a parallel mesh based on one of
>>>>>>>> several "partitionings" (collection of collections) stored with a
>>>>>>>> mesh, e.g. using the geometric volumes, material types, or true
>>>>>>>> partitions (generated by a partitioner).
>>>>>>> This is a problem we always had, though, didn't we?  I mean, with
>>>>>>> multiple instances per process, repartitioning to use a new
>>>>>>> instance would require copying the whole mesh, including
>>>>>>> adjacency, sets, tags, etc.  Do-able, but non-trivial, especially
>>>>>>> (say) tags with entity handle value.
>>>>>>> I don't have any conceptual issues in principle with the notion of
>>>>>>> multiple partitionings of the same entities.  How an
>>>>>>> implementation (or app) chooses to store the inactive partition is
>>>>>>> obviously going to be implementation-dependent (sets vs tags vs
>>>>>>> classification vs ???).  This potentially makes it tricky to
>>>>>>> define an API for switching between them.
>>>>>>> Also, what about the different, more subtle case where we -need-
>>>>>>> two partitions active at once?  One use case is a contact problem,
>>>>>>> where the 3D mesh and 2D surface are partitioned separately (I'm
>>>>>>> assuming this is done, to get better load balance?).  If you're
>>>>>>> also doing mesh adaptation as you go (likely), now those 2D faces
>>>>>>> on the interface have owners in the 3D partition (okay, some
>>>>>>> implementation don't represent them explicitly, but it's safe to
>>>>>>> say that a face interior to a part is going to be owned by that
>>>>>>> part...).  I see two alternatives.
>>>>>>> 1. The easiest way to handle this in the current interfaces, as
>>>>>>> far as parallel stuff goes, is to have two separate meshes (a 3D
>>>>>>> mesh and a 2D manifold mesh), with tags or iRel (or something
>>>>>>> similar) keeping the respective entities associated with each
>>>>>>> other.  This implies two iMesh instances per process, one each for
>>>>>>> the 3D and 2D manifold meshes, which isn't in conflict with other
>>>>>>> stuff in this thread.
>>>>>>> 2. You could also have two partitions active within a single
>>>>>>> instance. I'm not quite sure how this would look, even
>>>>>>> conceptually.  You'd have the full 3D mesh for the (3D) part, plus
>>>>>>> a bunch of extra faces (and verts, at least) for the bits of the
>>>>>>> 2D manifold mesh that isn't already resident.  The problem with
>>>>>>> this is that you've got to strictly segregate things so that
>>>>>>> iteration over (say) faces gets you what you wanted.  Yes, I know,
>>>>>>> sets will do this, but it seems cumbersome...
>>>>>>> I can't say I'm crazy about either of these alternatives, nor am I
>>>>>>> invested in either one.  Hopefully, someone can come up with
>>>>>>> something better.  For what's it's worth, I think this problem is
>>>>>>> reasonably (completely?) independent of whether in "normal" single
>>>>>>> mesh contexts we require exactly one part and iMesh instance per
>>>>>>> process.
>>>>>>> Having said that, I'll almost certainly be missing tomorrow's
>>>>>>> telecon. I will, of course, be happy to kibitz afterwards by
>>>>>>> email. :-)
>>>>>>> Carl
>>>> 
>>> 
>> 
>> 
> 
> --
> ================================================================
> "You will keep in perfect peace him whose mind is
>    steadfast, because he trusts in you."               Isaiah 26:3
> 
>               Tim Tautges            Argonne National Laboratory
>           (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>           phone: (608) 263-8485      1500 Engineering Dr.
>             fax: (608) 263-4499      Madison, WI 53706
> 
> 
>