itaps-parallel Proposal for handling queries with parts, sets, and partitions

[Tim Tautges]

Vitus Leung wrote:

>> I think it's the other way around: the part/partition resides in the 
>> mesh instance.  In parallel, there's an implicit relation between a 
>> partition on one processor and on another processor, such that both/all 
>> processors understand that their local partition information is a piece 
>> of a collection over all processors.  This model reduces to the trivial 
>> case for a single processor, both for partition(s) in mesh instances and 
>> parts in partitions.
>>
> 
> I thought that the mesh instance for a processor is what resides on that
> processor alone.  This would make it impossible for a mesh instance to
> contain a partition.
> 

The mesh instance is what resides on that processor, but there's also an 
implicit relation between instances and partitions when running in 
parallel.  Think of running the same mesh in serial then on 2 procs then 
on 4096.  You want the same boundary condition sets to exist over the 
whole mesh in each case, and in each case you have a partition, possibly 
with exactly the same number of parts in all cases.  In serial the 
instance happens to contain the partition and all the parts; in parallel 
each instance contains a partition set, understood to be the same as the 
partition sets on the other processors, and the parts which are 
local/assigned to that processor.  Similarly, each processor has a 
boundary condition set, understood to be the same as the boundary 
condition sets on other processors, and the entities which reside 
locally on that processor in that set.  If a given processor doesn't 
have any entities locally in that set, that processor may or may not 
have that (empty) set.  For MOAB, the application (parallel meshing, in 
this case) tells the parallel writer which tag[/value] combinations 
should be looked for to find sets spanning processors; thus, you can 
have an arbitrary number of sets, of multiple types, spanning or not 
spanning processors.  I imagine a reader might keep information about 
that around after reading, though I haven't implemented that yet.

- tim

>>> I propose that we think about both parts and mesh instances as coverings 
>>> of the global mesh (they are), and that we also think about parts, mesh
>>> instances, and the partition as containing -all- data about some 
>>> (topologically and/or geometrically compact subset) or all of the global 
>>> mesh (as opposed to entity sets, which are deliberately more selective). 
>>>  Yes, I know they're all collections of entities, but stick with me 
>>> while I justify why I think this semantic distinction is worthwhile. 
>>> Also yes, we haven't decided yet (have we?) whether a part will 
>>> officially contain everything or just the entities that were 
>>> partitioned; I don't -think- that any of what I'm about to propose is 
>>> adversely affected, either way.
>>>
>> I don't think the partition can be a covering of the whole mesh on a 
>> given processor.  Given a collection of 2d and 3d elements, you need to 
>> distinguish between the 2d elements that are just adjacent to objects in 
>> the partition and the 2d elements which are in fact objects in the 
>> partition.
>>
>> I did not see anything following which justifies the semantic 
>> distinction you're talking about.
>>
>>> Finally, it's worth noting that, while a partition contains many parts
>>> distributed over many processes, each process is associated with a
>>> unique partition (MPI communicator or equivalent) and each part is
>>> associated both with a unique process and a unique partition.  So
>>> specifying both part handle and partition handle is actually redundant.
>>>
>> Each part is associated with a unique partition and process, but each 
>> process is not associated with a unique partition.  There may be 
>> multiple partitions in use at any given time, and a given process may be 
>> responsible for one or more parts in each partition.  Take parallel 
>> contact detection for instance.  One partition is over volume elements, 
>> the other is over faces on the skin of those elements.  Processors 
>> participate in calculations for both partitions at different times.
>>
>> Specifying both a part and partition handle may not be strictly 
>> redundant, and in terms of implementation I don't think it's useful to 
>> to require that the partition be derivable from the part handle.  I do 
>> think that in most cases, an application will know the partition that's 
>> being dealt with.  For the few cases where a part is a member of 
>> multiple partitions (if we want to allow that, and I don't see a major 
>> reason why not to), there should be a function to get the partition(s) 
>> that a part is a member of.
>>
>>> Given that backgroun, I propose that we overload our current query
>>> functions so that any place where a mesh instance is currently usable, a
>>> part handle or partition handle is also usable, and continue to use
>>> either a bona fide entity set or the (global placeholder) root set as
>>> the second argument.  In this scenario, we would have, for instance:
>>>
>>>             / partition handle
>>> getNumOfType| mesh instance    , root set handle   , type, result, err )
>>>             \ part handle        entity set handle
>>>
>>> The other iMesh function that is obviously of this type is getNumOfTopo.
>>>
>>> A somewhat larger collection of functions would be able to take both
>>> mesh instances and part handles as the first argument, but not partition
>>> handles: getAllVtxCoords, getVtxCoordIndex, getEntities.
>>>
>>> My guess is that iterator functions will fall into the second category.
>>>
>>> Many of the set queries (number of sets, number of children, identities 
>>> of sets and children) are a bit more ambiguous here, to my surprise: 
>>> essentially, if you ask for the number of sets contained in a given set 
>>> for a particular part, you presumably mean the number of contained sets 
>>> that intersect that part.  While I don't have a problem with that 
>>> definition, I'm not sure it's an entirely straightforward one, either 
>>> conceptually or in implementation.  But I haven't thought about it in a 
>>> great deal of detail, either...  One way to finesse this would be to 
>>> create sets at the part level rather than the mesh instance level; then 
>>> all is easy again.
>>>
>> So again, I go back to asking: what are the core needs that prevent us 
>> from using sets as both parts and partitions?  The entity set mechanism 
>> was designed with this specific usage in mind.
>>
>> I know some haven't fully implemented sets, but is that going to be more 
>> difficult than the degree of interface changes being discussed here?
>>
>> The things that are missing in sets for use as partitions and parts are 
>> exactly the extensions we'll already need in sets for going to parallel, 
>> no matter what we choose for partition and part representation. 
>> Specifically, we'll need some notion of a correspondence between sets on 
>> different processors, to handle boundary conditions on mesh spanning 
>> processors.
>>
>> I'll make a diagram equivalent to Carl's today, maybe that will make 
>> things a bit clearer.
>>
>> - tim
>>
>>> Now, there are a couple of beneath-the-hood requirements for
>>> implementations here:
>>>
>>> 1.  All part handles must be unique, even in the presence of multiple
>>>     partitions.  Pointer-type handles will easily satisfy this;
>>>     integer-type handles may need to reserve some bits for partition ID
>>>     and some for part ID.
>>>
>>> 2.  In most (all?) implementations, finding the result for calls with
>>>     (part handle, entity set handle...) will require an implementation
>>>     to do some sort of intersection internally.  This will prove
>>>     especially challenging for iterator in the presence of mesh
>>>     modification and/or migration.
>>>
>>> I recognize also that this paradigm shift potentially ambiguates the use
>>> of calls like addEntToSet for adding entities to a part; we can either
>>> overload those functions (probably renamed to addEntToCollection) or
>>> create new addEntToPart functions (I think there are only going to be
>>> four of these: add/rmv single/array).
>>>
>>> Okay, so there's my proposal.  I'll now stand back and let people poke
>>> holes in the idea; that may not prove to be too difficult.
>>>
>>> Carl
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
> 
> 
> 

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706