itaps-parallel Notes from today's meeting

[Mark Shephard]

Tim,

I am sure that we can effectively implement contact, including the 
evolution of contact and inclusion of mesh adaptation if desired as I 
have indicated (a single partition of the mesh). The issue of needing to 
migrate things between parts during adaptation has nothing to do with 
it. Some form of dynamic load balancing is to account for any change 
(contact evolution or mesh adaptation) if one wants to continue to 
scale. Again, that is independent of the other questions.

Mark


tautges at mcs.anl.gov wrote:
> I'm not sure this is true. If I understand your amr method correctly, 
> you only adapt the interior of meshes, migrating a potion of an 
> interface to make that so.  Or, have you changed that?  Whereas with 
> parallel contact (at least as implemented by sandia, which I'm familiar 
> with), the face partition is completely unrelated to the region partition.
> 
> - tim
> 
> 
> On Apr 19, 2010, at 5:15 AM, Mark Shephard <shephard at scorec.rpi.edu> wrote:
> 
>> Two quick comments/questions:
>>
>> Is there a conference call today, Monday? If yes, what time?
>>
>> The issues of doing a contact problem in parallel are really much the 
>> say as doing mesh adaptivity in parallel with a form of nonmanifold 
>> model. The only new item past that of a 2-manifold situation is 
>> keeping  track of the contact face interactions. There is absolutely 
>> no problem doing this within in the concept of a single partition of 
>> the mesh. On the other hand, considering how to track an evolving 
>> contact interface through the use of multiple partitions of the mesh 
>> would actually add substantial complexity to the mesh up-date and 
>> dynamic load balancing processes.
>>
>> Mark
>>
>> Carl Ollivier-Gooch wrote:
>>> Tim Tautges wrote:
>>>> I think Jason's argument is important here: the easy things should 
>>>> be easy, and more complicated things possible.  I think this is 
>>>> related to Seegyoung's point about having a single API.  I think 
>>>> what this boils down to is an assertion that we should think 
>>>> primarily in terms of one iMesh Instance per process.  For those 
>>>> applications wanting to use multiple instances, that implies 
>>>> communication between the instances at least conceptually, if not in 
>>>> the implementation also.  What this boils down to even further is 
>>>> that Part and iMesh instance are synonymous.  The issue of an entity 
>>>> assigned to exactly one Part morphs to one of representation vs. 
>>>> ownership: an entity can be represented in more than one instance 
>>>> (e.g. faces on part boundaries, in an element-based partition), but 
>>>> can be owned by only one of those instances.
>>> Personally, I'm very much in favor of the part = instance 
>>> interpretation, for reasons already outlined.
>>> In addition, I can easily see how we'll be able to implement this, 
>>> whereas with multiple parts per instance, the easiest approach would 
>>> be for us, effectively, to have multiple of our current instances 
>>> inside a wrapper; yuck!
>>> At a higher level, this also makes the whole parts-as-special-sets 
>>> argument moot, saving us from possibly re-visiting that.  If this is 
>>> the way we decide to go, we'll want to do an audit of iMeshP 
>>> functions, because part handles will now be synonymous with iMesh 
>>> instance handles.  I suspect that a whole lot of functions either go 
>>> away or get simpler.  This is not a bad thing.
>>>> One of the reasons we initially thought in terms of multiple parts 
>>>> per process was to handle over-partitionings, where a process could 
>>>> be assigned multiple parts, e.g. for better load balancing, and 
>>>> flexibility in what's stored with a mesh or how it's stored.  I 
>>>> think that concept is an artifact of how a partitioning is stored 
>>>> with the mesh (or how the mesh is stored in files to reflect a 
>>>> partitioning).  From an application's point of view, in most cases 
>>>> it's still going to want to access the mesh from a single instance 
>>>> on a given process, no matter how that mesh was loaded initially.  
>>>> That's also how the mesh looks from most applications' points of 
>>>> view.  The use case of multiple parts, with shared interface 
>>>> entities duplicated, is mostly handled by having separate entities 
>>>> in the database, with special relations between them implying that 
>>>> they're the same.
>>> I can see ways around the over-partitioning problem, though.  Most 
>>> obviously, simply have multiple threads per processor, so that the 
>>> number of processes still matches the number of parts.  Or an app or 
>>> implementation (or even the interface) could specify how to merge 
>>> parts to get the right number at run time, as Tim hints at.
>>> Eventually, in the hyperparallel world that's coming, apps and 
>>> implementations may both have to be written to be multithreaded 
>>> shared-memory processes within a node (OpenMP?) with communication 
>>> between nodes (MPI?).  This isn't in conflict with the notion of 
>>> exactly one part (and iMesh instance) per process; it just requires 
>>> more careful programming of apps/services that use the interface.
>>>> One of the complications that arises from having one instance per 
>>>> process (and one part per instance) is how do you repartition, or 
>>>> partition in serial.  I think that concept is handled by the notion 
>>>> of initializing a given parallel mesh either a) from a collection of 
>>>> collections, or a partition of parts, stored with and loaded with 
>>>> the mesh, or b) communication from one instance to another, using 
>>>> parallel migration of entities.  Of course, a) is easily handled 
>>>> using sets, if you have them; if you don't, it's just as easily 
>>>> handled using multiple files.  It also naturally handles 
>>>> establishing a parallel mesh based on one of several "partitionings" 
>>>> (collection of collections) stored with a mesh, e.g. using the 
>>>> geometric volumes, material types, or true partitions (generated by 
>>>> a partitioner).
>>> This is a problem we always had, though, didn't we?  I mean, with 
>>> multiple instances per process, repartitioning to use a new instance 
>>> would require copying the whole mesh, including adjacency, sets, 
>>> tags, etc.  Do-able, but non-trivial, especially (say) tags with 
>>> entity handle value.
>>> I don't have any conceptual issues in principle with the notion of 
>>> multiple partitionings of the same entities.  How an implementation 
>>> (or app) chooses to store the inactive partition is obviously going 
>>> to be implementation-dependent (sets vs tags vs classification vs 
>>> ???).  This potentially makes it tricky to define an API for 
>>> switching between them.
>>> Also, what about the different, more subtle case where we -need- two 
>>> partitions active at once?  One use case is a contact problem, where 
>>> the 3D mesh and 2D surface are partitioned separately (I'm assuming 
>>> this is done, to get better load balance?).  If you're also doing 
>>> mesh adaptation as you go (likely), now those 2D faces on the 
>>> interface have owners in the 3D partition (okay, some implementation 
>>> don't represent them explicitly, but it's safe to say that a face 
>>> interior to a part is going to be owned by that part...).  I see two 
>>> alternatives.
>>> 1. The easiest way to handle this in the current interfaces, as far 
>>> as parallel stuff goes, is to have two separate meshes (a 3D mesh and 
>>> a 2D manifold mesh), with tags or iRel (or something similar) keeping 
>>> the respective entities associated with each other.  This implies two 
>>> iMesh instances per process, one each for the 3D and 2D manifold 
>>> meshes, which isn't in conflict with other stuff in this thread.
>>> 2. You could also have two partitions active within a single 
>>> instance. I'm not quite sure how this would look, even conceptually.  
>>> You'd have the full 3D mesh for the (3D) part, plus a bunch of extra 
>>> faces (and verts, at least) for the bits of the 2D manifold mesh that 
>>> isn't already resident.  The problem with this is that you've got to 
>>> strictly segregate things so that iteration over (say) faces gets you 
>>> what you wanted.  Yes, I know, sets will do this, but it seems 
>>> cumbersome...
>>> I can't say I'm crazy about either of these alternatives, nor am I 
>>> invested in either one.  Hopefully, someone can come up with 
>>> something better.  For what's it's worth, I think this problem is 
>>> reasonably (completely?) independent of whether in "normal" single 
>>> mesh contexts we require exactly one part and iMesh instance per 
>>> process.
>>> Having said that, I'll almost certainly be missing tomorrow's 
>>> telecon. I will, of course, be happy to kibitz afterwards by email. :-)
>>> Carl
>