itaps-parallel Notes from today's meeting

[tautges at mcs.anl.gov]

That sounds really hard, but ok.

- tim


On Apr 19, 2010, at 8:55 AM, Mark Shephard <shephard at scorec.rpi.edu>  
wrote:

> Tim,
>
> I am sure that we can effectively implement contact, including the  
> evolution of contact and inclusion of mesh adaptation if desired as  
> I have indicated (a single partition of the mesh). The issue of  
> needing to migrate things between parts during adaptation has  
> nothing to do with it. Some form of dynamic load balancing is to  
> account for any change (contact evolution or mesh adaptation) if one  
> wants to continue to scale. Again, that is independent of the other  
> questions.
>
> Mark
>
>
> tautges at mcs.anl.gov wrote:
>> I'm not sure this is true. If I understand your amr method  
>> correctly, you only adapt the interior of meshes, migrating a  
>> potion of an interface to make that so.  Or, have you changed  
>> that?  Whereas with parallel contact (at least as implemented by  
>> sandia, which I'm familiar with), the face partition is completely  
>> unrelated to the region partition.
>> - tim
>> On Apr 19, 2010, at 5:15 AM, Mark Shephard  
>> <shephard at scorec.rpi.edu> wrote:
>>> Two quick comments/questions:
>>>
>>> Is there a conference call today, Monday? If yes, what time?
>>>
>>> The issues of doing a contact problem in parallel are really much  
>>> the say as doing mesh adaptivity in parallel with a form of  
>>> nonmanifold model. The only new item past that of a 2-manifold  
>>> situation is keeping  track of the contact face interactions.  
>>> There is absolutely no problem doing this within in the concept of  
>>> a single partition of the mesh. On the other hand, considering how  
>>> to track an evolving contact interface through the use of multiple  
>>> partitions of the mesh would actually add substantial complexity  
>>> to the mesh up-date and dynamic load balancing processes.
>>>
>>> Mark
>>>
>>> Carl Ollivier-Gooch wrote:
>>>> Tim Tautges wrote:
>>>>> I think Jason's argument is important here: the easy things  
>>>>> should be easy, and more complicated things possible.  I think  
>>>>> this is related to Seegyoung's point about having a single API.   
>>>>> I think what this boils down to is an assertion that we should  
>>>>> think primarily in terms of one iMesh Instance per process.  For  
>>>>> those applications wanting to use multiple instances, that  
>>>>> implies communication between the instances at least  
>>>>> conceptually, if not in the implementation also.  What this  
>>>>> boils down to even further is that Part and iMesh instance are  
>>>>> synonymous.  The issue of an entity assigned to exactly one Part  
>>>>> morphs to one of representation vs. ownership: an entity can be  
>>>>> represented in more than one instance (e.g. faces on part  
>>>>> boundaries, in an element-based partition), but can be owned by  
>>>>> only one of those instances.
>>>> Personally, I'm very much in favor of the part = instance  
>>>> interpretation, for reasons already outlined.
>>>> In addition, I can easily see how we'll be able to implement  
>>>> this, whereas with multiple parts per instance, the easiest  
>>>> approach would be for us, effectively, to have multiple of our  
>>>> current instances inside a wrapper; yuck!
>>>> At a higher level, this also makes the whole parts-as-special- 
>>>> sets argument moot, saving us from possibly re-visiting that.  If  
>>>> this is the way we decide to go, we'll want to do an audit of  
>>>> iMeshP functions, because part handles will now be synonymous  
>>>> with iMesh instance handles.  I suspect that a whole lot of  
>>>> functions either go away or get simpler.  This is not a bad thing.
>>>>> One of the reasons we initially thought in terms of multiple  
>>>>> parts per process was to handle over-partitionings, where a  
>>>>> process could be assigned multiple parts, e.g. for better load  
>>>>> balancing, and flexibility in what's stored with a mesh or how  
>>>>> it's stored.  I think that concept is an artifact of how a  
>>>>> partitioning is stored with the mesh (or how the mesh is stored  
>>>>> in files to reflect a partitioning).  From an application's  
>>>>> point of view, in most cases it's still going to want to access  
>>>>> the mesh from a single instance on a given process, no matter  
>>>>> how that mesh was loaded initially.  That's also how the mesh  
>>>>> looks from most applications' points of view.  The use case of  
>>>>> multiple parts, with shared interface entities duplicated, is  
>>>>> mostly handled by having separate entities in the database, with  
>>>>> special relations between them implying that they're the same.
>>>> I can see ways around the over-partitioning problem, though.   
>>>> Most obviously, simply have multiple threads per processor, so  
>>>> that the number of processes still matches the number of parts.   
>>>> Or an app or implementation (or even the interface) could specify  
>>>> how to merge parts to get the right number at run time, as Tim  
>>>> hints at.
>>>> Eventually, in the hyperparallel world that's coming, apps and  
>>>> implementations may both have to be written to be multithreaded  
>>>> shared-memory processes within a node (OpenMP?) with  
>>>> communication between nodes (MPI?).  This isn't in conflict with  
>>>> the notion of exactly one part (and iMesh instance) per process;  
>>>> it just requires more careful programming of apps/services that  
>>>> use the interface.
>>>>> One of the complications that arises from having one instance  
>>>>> per process (and one part per instance) is how do you  
>>>>> repartition, or partition in serial.  I think that concept is  
>>>>> handled by the notion of initializing a given parallel mesh  
>>>>> either a) from a collection of collections, or a partition of  
>>>>> parts, stored with and loaded with the mesh, or b) communication  
>>>>> from one instance to another, using parallel migration of  
>>>>> entities.  Of course, a) is easily handled using sets, if you  
>>>>> have them; if you don't, it's just as easily handled using  
>>>>> multiple files.  It also naturally handles establishing a  
>>>>> parallel mesh based on one of several  
>>>>> "partitionings" (collection of collections) stored with a mesh,  
>>>>> e.g. using the geometric volumes, material types, or true  
>>>>> partitions (generated by a partitioner).
>>>> This is a problem we always had, though, didn't we?  I mean, with  
>>>> multiple instances per process, repartitioning to use a new  
>>>> instance would require copying the whole mesh, including  
>>>> adjacency, sets, tags, etc.  Do-able, but non-trivial, especially  
>>>> (say) tags with entity handle value.
>>>> I don't have any conceptual issues in principle with the notion  
>>>> of multiple partitionings of the same entities.  How an  
>>>> implementation (or app) chooses to store the inactive partition  
>>>> is obviously going to be implementation-dependent (sets vs tags  
>>>> vs classification vs ???).  This potentially makes it tricky to  
>>>> define an API for switching between them.
>>>> Also, what about the different, more subtle case where we -need-  
>>>> two partitions active at once?  One use case is a contact  
>>>> problem, where the 3D mesh and 2D surface are partitioned  
>>>> separately (I'm assuming this is done, to get better load  
>>>> balance?).  If you're also doing mesh adaptation as you go  
>>>> (likely), now those 2D faces on the interface have owners in the  
>>>> 3D partition (okay, some implementation don't represent them  
>>>> explicitly, but it's safe to say that a face interior to a part  
>>>> is going to be owned by that part...).  I see two alternatives.
>>>> 1. The easiest way to handle this in the current interfaces, as  
>>>> far as parallel stuff goes, is to have two separate meshes (a 3D  
>>>> mesh and a 2D manifold mesh), with tags or iRel (or something  
>>>> similar) keeping the respective entities associated with each  
>>>> other.  This implies two iMesh instances per process, one each  
>>>> for the 3D and 2D manifold meshes, which isn't in conflict with  
>>>> other stuff in this thread.
>>>> 2. You could also have two partitions active within a single  
>>>> instance. I'm not quite sure how this would look, even  
>>>> conceptually.  You'd have the full 3D mesh for the (3D) part,  
>>>> plus a bunch of extra faces (and verts, at least) for the bits of  
>>>> the 2D manifold mesh that isn't already resident.  The problem  
>>>> with this is that you've got to strictly segregate things so that  
>>>> iteration over (say) faces gets you what you wanted.  Yes, I  
>>>> know, sets will do this, but it seems cumbersome...
>>>> I can't say I'm crazy about either of these alternatives, nor am  
>>>> I invested in either one.  Hopefully, someone can come up with  
>>>> something better.  For what's it's worth, I think this problem is  
>>>> reasonably (completely?) independent of whether in "normal"  
>>>> single mesh contexts we require exactly one part and iMesh  
>>>> instance per process.
>>>> Having said that, I'll almost certainly be missing tomorrow's  
>>>> telecon. I will, of course, be happy to kibitz afterwards by  
>>>> email. :-)
>>>> Carl
>