itaps-parallel Tag communication operations

[Tim Tautges]

Carl Ollivier-Gooch wrote:
> Tim Tautges wrote:
>> Hi all,
>>   Regarding tag communication operations, the goal is to provide in 
>> the interface functions that most parallel applications are likely to 
>> need.  One functionality that comes to mind right away is exchanging 
>> tag information on shared and/or ghosted entities.  We could get fancy 
>> about it, getting more into what kinds of operations would be provided 
>> (min/max, average, etc.), but at this point I think it would be better 
>> to keep it simple.  So, I'd propose two functions:
>>
>> sendTagsShared: communicate specified tag to all processors sharing 
>> entities owned by this processor
>>
>> sendTagsGhosted: same, but for ghosted entities
>>
>> The particular syntax for these functions can come later, I just want 
>> to get some comments on the general idea.  If applications wanted some 
>> sort of higher-level behavior, like the operations mentioned above, 
>> additional tags could be used to implement those fairly easily.
> 
> Tim,
> 
> My interpretation of what you propose is that, for a given tag name or 
> handle, each process (as opposed to each part) will communicate the 
> values of that tag on all entities with duplicates elsewhere.
> 
> Comments/questions:
> 
> 1.  Just to confirm, you mean this to be a push from the owned copy of 
> the entity (which, one could argue, is the only one with a right to 
> change tag data on the entity anyway...).
> 

A push using two-sided communication (i.e. would like to avoid 1-sided 
communication for now).

> 2.  What about implementations that work part-by-part, even when there 
> are multiple parts in a process?  Admittedly the communication overhead 
> is trivial for this case, but part to part updates of tag data within a 
> process are going to matter to some apps. (Ken, am I right there?)
> 

I should have written the descriptions in terms of parts rather than 
processes (though I still assert that most applications' need for 
multiple part per processor/core has more to do with difficulty of 
current data storage mechanisms and partitioning tools than a need based 
on other things).

> 3.  This call implies either an ability to reverse-lookup entities by 
> tag handle or an iteration over all entities (of all types) or some 
> clever call syntax.  Or at least, I can't imagine any alternatives that 
> don't....
> 

I'm assuming it's easy to get which entities are shared with other 
parts, and the application should know when a given tag exists on shared 
entities (although maybe we need a type/topology argument to be more 
specific?)

> 4.  Do you envision these as collective, blocking calls with everything 
> done before the functions return?  Or as non-blocking send / blocking 
> receive pairs?
> 

We need at least blocking calls, though this doesn't necessarily imply 
that every communication call is blocking (in fact, I'd expect that 
within these calls non-blocking calls would be used, but the return from 
the entire function would block).  These wouldn't necessarily be 
collective, they'd just have to match in terms of which other parts a 
given part communicated with.  Though maybe we should also have a 
part/list of parts argument?

> 5.  Somewhat off the direct topic, but what are the merits of 
> implementing the global reduce sorts of operations as a service on top 
> of the interface?
> 

I think that would be doable.

> 6.  Also somewhat off topic, and probably one to which a reply directly 
> to me rather than to the list is appropriate: I can't see how one would 
> efficiently implement both dense tags (like solution data) and sparse 
> tags (like which vertices need smoothing).  Dense tags seem to scream to 
> be stored as arrays, which implies an explicit numbering of entities, 
> while sparse tags seem more like they should be stored as a map or hash 
> on entity handle, or a linked list attached to the entity.  Anyone who 
> has a good solution for handling both types efficiently, I'd love to 
> hear it.
> 

This is precisely what MOAB does.  Dense tags are stored as arrays, 
blocked by series of contiguous handles, sparse tags as a map of 
handle/value pairs.  This performs well in both memory and speed.  It's 
one of the important reasons C++ object-based storage isn't used in MOAB.

- tim

> Cheers,
> Carl
> 

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706