itaps-parallel Notes from today's meeting

[Tim Tautges]

The issue addressed in the work I'm citing had to do with load balancing of the surface vs. volume calculation, where 
surface here means geometric surface, not interprocessor boundary.  In that work, they needed to check for contact 
frequently (maybe even every timestep?), but keep the volumetric partition also, so migration was out of the question. 
That's why I call it different from migration before/after adaptation.  I'm sure there are multiple ways of implementing 
it, though.

- tim


Mark Shephard wrote:
> It is not hard at all. In fact it is quite straight forward and clean to 
> add an implementation that already deals with fine level evolution of 
> the simulation model.
> 
> tautges at mcs.anl.gov wrote:
>> That sounds really hard, but ok.
>>
>> - tim
>>
>>
>> On Apr 19, 2010, at 8:55 AM, Mark Shephard <shephard at scorec.rpi.edu> 
>> wrote:
>>
>>> Tim,
>>>
>>> I am sure that we can effectively implement contact, including the 
>>> evolution of contact and inclusion of mesh adaptation if desired as I 
>>> have indicated (a single partition of the mesh). The issue of needing 
>>> to migrate things between parts during adaptation has nothing to do 
>>> with it. Some form of dynamic load balancing is to account for any 
>>> change (contact evolution or mesh adaptation) if one wants to 
>>> continue to scale. Again, that is independent of the other questions.
>>>
>>> Mark
>>>
>>>
>>> tautges at mcs.anl.gov wrote:
>>>> I'm not sure this is true. If I understand your amr method 
>>>> correctly, you only adapt the interior of meshes, migrating a potion 
>>>> of an interface to make that so.  Or, have you changed that?  
>>>> Whereas with parallel contact (at least as implemented by sandia, 
>>>> which I'm familiar with), the face partition is completely unrelated 
>>>> to the region partition.
>>>> - tim
>>>> On Apr 19, 2010, at 5:15 AM, Mark Shephard <shephard at scorec.rpi.edu> 
>>>> wrote:
>>>>> Two quick comments/questions:
>>>>>
>>>>> Is there a conference call today, Monday? If yes, what time?
>>>>>
>>>>> The issues of doing a contact problem in parallel are really much 
>>>>> the say as doing mesh adaptivity in parallel with a form of 
>>>>> nonmanifold model. The only new item past that of a 2-manifold 
>>>>> situation is keeping  track of the contact face interactions. There 
>>>>> is absolutely no problem doing this within in the concept of a 
>>>>> single partition of the mesh. On the other hand, considering how to 
>>>>> track an evolving contact interface through the use of multiple 
>>>>> partitions of the mesh would actually add substantial complexity to 
>>>>> the mesh up-date and dynamic load balancing processes.
>>>>>
>>>>> Mark
>>>>>
>>>>> Carl Ollivier-Gooch wrote:
>>>>>> Tim Tautges wrote:
>>>>>>> I think Jason's argument is important here: the easy things 
>>>>>>> should be easy, and more complicated things possible.  I think 
>>>>>>> this is related to Seegyoung's point about having a single API.  
>>>>>>> I think what this boils down to is an assertion that we should 
>>>>>>> think primarily in terms of one iMesh Instance per process.  For 
>>>>>>> those applications wanting to use multiple instances, that 
>>>>>>> implies communication between the instances at least 
>>>>>>> conceptually, if not in the implementation also.  What this boils 
>>>>>>> down to even further is that Part and iMesh instance are 
>>>>>>> synonymous.  The issue of an entity assigned to exactly one Part 
>>>>>>> morphs to one of representation vs. ownership: an entity can be 
>>>>>>> represented in more than one instance (e.g. faces on part 
>>>>>>> boundaries, in an element-based partition), but can be owned by 
>>>>>>> only one of those instances.
>>>>>> Personally, I'm very much in favor of the part = instance 
>>>>>> interpretation, for reasons already outlined.
>>>>>> In addition, I can easily see how we'll be able to implement this, 
>>>>>> whereas with multiple parts per instance, the easiest approach 
>>>>>> would be for us, effectively, to have multiple of our current 
>>>>>> instances inside a wrapper; yuck!
>>>>>> At a higher level, this also makes the whole parts-as-special-sets 
>>>>>> argument moot, saving us from possibly re-visiting that.  If this 
>>>>>> is the way we decide to go, we'll want to do an audit of iMeshP 
>>>>>> functions, because part handles will now be synonymous with iMesh 
>>>>>> instance handles.  I suspect that a whole lot of functions either 
>>>>>> go away or get simpler.  This is not a bad thing.
>>>>>>> One of the reasons we initially thought in terms of multiple 
>>>>>>> parts per process was to handle over-partitionings, where a 
>>>>>>> process could be assigned multiple parts, e.g. for better load 
>>>>>>> balancing, and flexibility in what's stored with a mesh or how 
>>>>>>> it's stored.  I think that concept is an artifact of how a 
>>>>>>> partitioning is stored with the mesh (or how the mesh is stored 
>>>>>>> in files to reflect a partitioning).  From an application's point 
>>>>>>> of view, in most cases it's still going to want to access the 
>>>>>>> mesh from a single instance on a given process, no matter how 
>>>>>>> that mesh was loaded initially.  That's also how the mesh looks 
>>>>>>> from most applications' points of view.  The use case of multiple 
>>>>>>> parts, with shared interface entities duplicated, is mostly 
>>>>>>> handled by having separate entities in the database, with special 
>>>>>>> relations between them implying that they're the same.
>>>>>> I can see ways around the over-partitioning problem, though.  Most 
>>>>>> obviously, simply have multiple threads per processor, so that the 
>>>>>> number of processes still matches the number of parts.  Or an app 
>>>>>> or implementation (or even the interface) could specify how to 
>>>>>> merge parts to get the right number at run time, as Tim hints at.
>>>>>> Eventually, in the hyperparallel world that's coming, apps and 
>>>>>> implementations may both have to be written to be multithreaded 
>>>>>> shared-memory processes within a node (OpenMP?) with communication 
>>>>>> between nodes (MPI?).  This isn't in conflict with the notion of 
>>>>>> exactly one part (and iMesh instance) per process; it just 
>>>>>> requires more careful programming of apps/services that use the 
>>>>>> interface.
>>>>>>> One of the complications that arises from having one instance per 
>>>>>>> process (and one part per instance) is how do you repartition, or 
>>>>>>> partition in serial.  I think that concept is handled by the 
>>>>>>> notion of initializing a given parallel mesh either a) from a 
>>>>>>> collection of collections, or a partition of parts, stored with 
>>>>>>> and loaded with the mesh, or b) communication from one instance 
>>>>>>> to another, using parallel migration of entities.  Of course, a) 
>>>>>>> is easily handled using sets, if you have them; if you don't, 
>>>>>>> it's just as easily handled using multiple files.  It also 
>>>>>>> naturally handles establishing a parallel mesh based on one of 
>>>>>>> several "partitionings" (collection of collections) stored with a 
>>>>>>> mesh, e.g. using the geometric volumes, material types, or true 
>>>>>>> partitions (generated by a partitioner).
>>>>>> This is a problem we always had, though, didn't we?  I mean, with 
>>>>>> multiple instances per process, repartitioning to use a new 
>>>>>> instance would require copying the whole mesh, including 
>>>>>> adjacency, sets, tags, etc.  Do-able, but non-trivial, especially 
>>>>>> (say) tags with entity handle value.
>>>>>> I don't have any conceptual issues in principle with the notion of 
>>>>>> multiple partitionings of the same entities.  How an 
>>>>>> implementation (or app) chooses to store the inactive partition is 
>>>>>> obviously going to be implementation-dependent (sets vs tags vs 
>>>>>> classification vs ???).  This potentially makes it tricky to 
>>>>>> define an API for switching between them.
>>>>>> Also, what about the different, more subtle case where we -need- 
>>>>>> two partitions active at once?  One use case is a contact problem, 
>>>>>> where the 3D mesh and 2D surface are partitioned separately (I'm 
>>>>>> assuming this is done, to get better load balance?).  If you're 
>>>>>> also doing mesh adaptation as you go (likely), now those 2D faces 
>>>>>> on the interface have owners in the 3D partition (okay, some 
>>>>>> implementation don't represent them explicitly, but it's safe to 
>>>>>> say that a face interior to a part is going to be owned by that 
>>>>>> part...).  I see two alternatives.
>>>>>> 1. The easiest way to handle this in the current interfaces, as 
>>>>>> far as parallel stuff goes, is to have two separate meshes (a 3D 
>>>>>> mesh and a 2D manifold mesh), with tags or iRel (or something 
>>>>>> similar) keeping the respective entities associated with each 
>>>>>> other.  This implies two iMesh instances per process, one each for 
>>>>>> the 3D and 2D manifold meshes, which isn't in conflict with other 
>>>>>> stuff in this thread.
>>>>>> 2. You could also have two partitions active within a single 
>>>>>> instance. I'm not quite sure how this would look, even 
>>>>>> conceptually.  You'd have the full 3D mesh for the (3D) part, plus 
>>>>>> a bunch of extra faces (and verts, at least) for the bits of the 
>>>>>> 2D manifold mesh that isn't already resident.  The problem with 
>>>>>> this is that you've got to strictly segregate things so that 
>>>>>> iteration over (say) faces gets you what you wanted.  Yes, I know, 
>>>>>> sets will do this, but it seems cumbersome...
>>>>>> I can't say I'm crazy about either of these alternatives, nor am I 
>>>>>> invested in either one.  Hopefully, someone can come up with 
>>>>>> something better.  For what's it's worth, I think this problem is 
>>>>>> reasonably (completely?) independent of whether in "normal" single 
>>>>>> mesh contexts we require exactly one part and iMesh instance per 
>>>>>> process.
>>>>>> Having said that, I'll almost certainly be missing tomorrow's 
>>>>>> telecon. I will, of course, be happy to kibitz afterwards by 
>>>>>> email. :-)
>>>>>> Carl
>>>
>>
> 
> 

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706