itaps-parallel Re: Parallel Interface philosophical comments

[Tim Tautges]

Onkar Sahni wrote:
>>> For clarification, by operations on entity sets I meant set-specific
>>> operations/functions, like iMesh_addEntToSet, iMesh_rmvEntFromSet,
>>> iMesh_subtract, iMesh_intersect, iMesh_unite etc. My understanding, from
>>> the last boot camp, is that these functions will not be enabled to
>>> accept
>>> part-handles.
>> I'm fine with that for set booleans, but for remove/add, I'm not sure.
>> How will a partitioner go about building partition sets?  And, if you
>> have things like part edges and part vertices (fig 3 in your pmodel1.3
>> document), why not use the set parent/child relation functions to
>> retrieve those relations instead of having new functions for that?
>>
> 
> I see above comments at two levels, first is concerning a part (is it a
> set?) and second is accounting inter-part boundaries like part edges, part
> verticies etc. (is it also a set?). In our implementation, both are not
> entity-sets.
> 
> Now, coming to the question why not use set parent/child relation
> functions (or why not use sets for parts and inter-part boundaries); there
> are multiple reasons and I will try to give a simple but necessary one.
> Consider two questions (that are elaborated in later part of email if
> someone is interested):
> 
> (a) For a giver part, which are the mesh entities residing on a inter-part
> boundary between say two parts (typically asked by applications like flow
> solver).
> (b) For a specific mesh entity residing on a inter-part boundary, what are
> its residence parts (typically asked by applications like mesh
> modification software).
> 
> It basically boils down to classification and reverse-classification (as
> we discussed in our last boot camp). If we only have
> reverse-classification question (b) becomes expensive and inconvenient to
> answer (for problems of practical interest, that would eventually be
> petascale).
> 

Yes.  To re-iterate, we want to allow for both and require neither. 
However, in the case where something is done, I think we should use 
existing functions to do it.  That doesn't mean you have to implement it 
that way underneath.

Instead of classification and reverse classification terms, I wish we'd 
use entity-based and set-based; the classification term has a previous 
usage implying geometry, which is confusing, at least to me.  I know 
saying set-based may imply to others that entity sets are used to 
implement it, though; maybe we need to find another, neutral term 
(collection?).  Carl, I don't want to hear anything about Fred or George.

So, for entity-based things, I think we should use existing tag 
functions, either with a conventional tag name, or an 
application-specified tag name.  For set-based things, we should use the 
existing set functions.  This follows well-known principles in API 
design (for example, see http://lcsd05.cs.tamu.edu/slides/keynote.pdf 
and http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=488).

> For (re-)partitioning of meshes, my understanding is that we would call
> partitioner lib. (like Zoltan) that will provide us destination part ids
> for part objects (which are typically regions for mesh on 3D manifold).
> Then we would perform mesh migration to create/fill/form the parts. Now,
> when a partitioner lib. (Zoltan) is called I am assuming we already have
> the mesh to start with (in case its initial partitioning step then there
> is only one part to begin with or in case if its re-partitioning step then
> there could be on one or more parts in the start). After call to
> partitioned lib., i.e., during mesh migration, mesh entities will be
> created (and some deleted) from part(s) as required. I agree with Carl's
> point 6 on mesh modification in his submission, which is somewhat related
> to this discussion.

We also want to allow the construction of a partitioning tool above the 
level of iMesh, such as the itaps-zoltan tool that Vitus has written. 
This tools is an iMesh application which grabs the mesh graph 
information, calls Zoltan to generate the partition, and embeds that 
partition into iMesh.  A couple important principles emerge when 
considering that example, in reference to your comments:

a) You might want to do a static partition, in parallel, starting with 
some other partitioning (e.g. material definitions or geometric topology 
groupings).  This is different than dynamic partitioning, where an 
existing parallel application wants to re-partition and migrate its 
data.  In this case, the partitioner still needs to write the partition 
onto the mesh, preferably as a tool sitting on top of iMesh.

b) You might want to write multiple partitions onto the same mesh, or 
consider different collections of coverings on a given mesh.  For 
example, you might not be able to load the mesh on a single processor, 
so you use a coarse partition/covering to load the mesh in parallel, 
then call a parallel static partitioner.

To handle these cases, you're going to need to extend the current 
proposed spec to handle multiple partitions.  The current spec for sets 
and tags, though, will handle this just fine.

> 
> Open question is how much control do we want to provide the
> user/application during (re-)partitioning step (say, do we want the
> user/application to do all the somewhat tedious job of mesh migration
> and/or we allow an input of destination part ids and we internally perform
> mesh migration), again depends who the user/application is. For cases with
> multi-parts per processor/process, we can go into further details when
> necessary.

I think we need to separate the notion of putting a mesh entity in a 
different part and migrating mesh to a different processor (I think the 
latter could be phrased as synching the parallel mesh layout to a 
collection of parts).

> 
>> I was referring to your statement below, "For existing functions we can
>> support compatibility with part-handles but main concern is for new ones
>> (which doesn't do existing/same things).".  Maybe by "new ones" you
>> meant the functions for this parallel interface?
>>
> 
> Yes, by "new ones" I meant functions for this parallel interface (which we
> are trying to define).
> 
> ---------------------------------
> Questions on classification and reverse-classification elaborated (if
> someone is interested):
> 
> (a) Consider a case where user (doing flow computations) wants to create
> communication trace between two parts for a flow solver (basically to see
> which dofs need to talk to each other between two parts), so user asks for
> mesh entities (which can be translated to dofs) that reside on both parts
> at inter-part boundaries (for simplicity consider that there is no
> ghosting involved). Basically we can provide a mechanism based on sets
> where the set contains all these entities (which we referred as
> reverse-classification in boot camp).
> 
> (b) Now, consider a different task where user (who is performing mesh
> modifications) wants to know residence parts for a particular mesh entity
> residing on a inter-part boundary. Considering there is only
> reverse-classification available (through sets), the way I think user can
> get this information is to first figure out the set which contains this
> mesh entity and then query information about that set (which contains the
> mesh entity). Now, in real cases involving unstructured meshes on say 100
> or 1000 parts, even by very modest estimates there will be say 50 sets for
> all inter-part boundaries it would be quite expensive and inconvenient for
> user to figure this out (now number of sets providing all inter-part
> boundaries depends on how things are defined but in any case the
> complexity involved will be significant if followed through
> reverse-classification approach).

Absolutely.  This is analogous to iRel, where you want the option of 
relating both entities and sets to entities or sets, for efficiency. 
It's a memory vs. speed tradeoff, and the application has to be able to 
choose.  This is another example of where our existing tag and set 
functionality would help us, since we can tag both entities and sets.

- tim

> ---------------------------------
> 
> - Onkar
> 
>>>>> (c) The new interface functions, we need to define for parts (to
>>>>> support
>>>>> parallel requirements/functionalities), will take part-handles as
>>>>> input
>>>>> and not any set-handles (if some implementation wants to use
>>>>> entity-sets
>>>>> for parts they can have part-handle and entity-set-handle as one and
>>>>> the
>>>>> same).
>>>> I agree that it will have to be an error if an application passes in a
>>>> set handle representing a set and not a part to one of the functions
>>>> expecting a part handle.  It'll have to be up to the implementation to
>>>> decide how to do that (using casting, looking for the tag designating
>>>> part-type sets, etc.)
>>> I guess I am not sure what you mean by set-handle not representing a
>>> part
>>> (or part-type sets). In our implementation there will be no set-handle
>>> representing a part (or part-type sets) rather there will be
>>> part-handle.
>>> I think there is a strong distinction between these two things, i.e.,
>>> set/set-handle and part/part-handle (as a part need not necessarily be a
>>> set in an implementation so why call part-type set or set-handle
>>> representing a part).
>>>
>> Sorry, I should have said "part handle" or "set handle" instead of
>> part-type set.
>>
>> - tim
>>
>>> - Onkar
>>>
>>>> - tim
>>>>
>>>>>> iRel is for relating data between different interfaces.  As I
>>>>>> understand
>>>>>> it, we are building parallel functionality as part of iMesh.
>>>>>> Therefore,
>>>>>> we shouldn't need to use iRel for this.
>>>>>>
>>>>>>> There are a number of groups that use classification as the primary
>>>>>>> mode
>>>>>>> of interaction for doing things similar to how the RPI group does.
>>>>>>> Just
>>>>>>> as we would not want you to have redo all your parallel work to do
>>>>>>> things strictly from a classification perspective, we do not expect
>>>>>>> that
>>>>>>> the ITAPS parallel interface will require RPI to re-do all its
>>>>>>> parallel
>>>>>>> adaptive work (years worth of work dating back to the mid-90's when
>>>>>>> we
>>>>>>> were the first to do it) that is being used for SciDAC applications
>>>>>>> to
>>>>>>> fit with a specific one way of doing things.
>>>>>>>
>>>>>>> By the way, some feel the classification is a simple and powerful
>>>>>>> way
>>>>>>> to
>>>>>>> do things and mesh sets are actually substantial extra overhead to a
>>>>>>> well defined model of using classification. In my experience, I
>>>>>>> would
>>>>>>> love to have both classification and reverse classification at all
>>>>>>> times, since depending on the question being asked, one is a faster
>>>>>>> way
>>>>>>> to answer it. However, due to the extra data and headache of keeping
>>>>>>> things correctly set, most I know tend to focus on one and use the
>>>>>>> other
>>>>>>> temporary manner when its advantageous.
>>>>>>>
>>>>>> I'm not advocating that we do things only with reverse
>>>>>> classification,
>>>>>> I
>>>>>> agree we need to be able to go both ways.  I'm just arguing that if
>>>>>> we
>>>>>> do pass part handles through the interface, then we operate on those
>>>>>> part handles using existing functions, rather than make new ones that
>>>>>> do
>>>>>> virtually the same thing.
>>>>> For existing functions we can support compatibility with part-handles
>>>>> but
>>>>> main concern is for new ones (which doesn't do existing/same things).
>>>>>
>>>>> - Onkar
>>>>>
>>>>>
>>>>>> - tim
>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>> Tim Tautges wrote:
>>>>>>>> Onkar Sahni wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Here are some of my questions and comments on ITAPS parallel
>>>>>>>>> interface:
>>>>>>>>>
>>>>>>>>> 1) Is there a difference between "instance" and "handle" in ITAPS
>>>>>>>>> terminology (or are there any definition for these)? (I may have
>>>>>>>>> missed it
>>>>>>>>> if this was resolved/defined in previous ITAPS interfaces like
>>>>>>>>> iMesh).
>>>>>>>>>
>>>>>>>> We use the generic term "handle" as an identifier for entities,
>>>>>>>> entity
>>>>>>>> sets, the interface itself, and tags.  In our C headers, these are
>>>>>>>> typedef'd to void*.  There are also iterators, which we haven't
>>>>>>>> really
>>>>>>>> called handles, but could be referred to that way.
>>>>>>>>
>>>>>>>>> 2) For situations where we want to support multi-parts per
>>>>>>>>> processor/process:
>>>>>>>>> (a) How do we want to distinguish between processor/process level
>>>>>>>>> functionalities against part level ones (keeping user- and
>>>>>>>>> implementation-convenience and efficiency in mind)? Ones which
>>>>>>>>> would
>>>>>>>>> fall
>>>>>>>>> under processor level are, like list of parts on processor,
>>>>>>>>> MPI_Rank
>>>>>>>>> etc.,
>>>>>>>>> whereas ones which would fall under part level are, like
>>>>>>>>> neighboring
>>>>>>>>> parts
>>>>>>>>> (not procs.) etc. Further, there could be ones where users wants
>>>>>>>>> to
>>>>>>>>> get/iterate mesh entities (and get adjacencies) at a processor
>>>>>>>>> and/or
>>>>>>>>> part
>>>>>>>>> level.
>>>>>>>> Processor-level queries are on the iMesh instance, part-level
>>>>>>>> queries
>>>>>>>> are on the entity set representing the part.  Our iterator
>>>>>>>> functions
>>>>>>>> operate on both (with processor-level queries simply using the root
>>>>>>>> set).
>>>>>>>>
>>>>>>>>> (b) Assuming a part-handle is supported, how can user/application
>>>>>>>>> get
>>>>>>>>> access to it (say on a processor where part resides)? Basically
>>>>>>>>> will
>>>>>>>>> there
>>>>>>>>> be a way to directly access the part-handle (like part iterator at
>>>>>>>>> iProcParts level) or user has to access it through tags/tag names,
>>>>>>>>> in
>>>>>>>>> which case user has to figure out what is the tag/tag name for a
>>>>>>>>> particular part (and who defines these tags/tag names).
>>>>>>>> I advocate having both: a conventional tag given to parts, so that
>>>>>>>> you
>>>>>>>> can find sets that are parts by looking for sets with that tag; and
>>>>>>>> also having a convenience function (as part of the parallel
>>>>>>>> extensions
>>>>>>>> to iMesh) which returns all the parts in the given instance (i.e.
>>>>>>>> on
>>>>>>>> the processor).  Note an important capability here for this
>>>>>>>> function
>>>>>>>> is to pass in an optional tag name/handle indicating the partition
>>>>>>>> you're asking about.  That allows multiple partitions to co-exist,
>>>>>>>> which is important for some applications.
>>>>>>>>
>>>>>>>>> 3) How general we want to be in interface for different data
>>>>>>>>> models/implementations (as in future we may want to
>>>>>>>>> incorporate/fit
>>>>>>>>> other
>>>>>>>>> implementations), especially between serial and parallel data. Two
>>>>>>>>> possibilities for data models I can think of are: (a) one (serial)
>>>>>>>>> data
>>>>>>>>> model is enhanced to store parallel information and (b) serial one
>>>>>>>>> is
>>>>>>>>> supplemented by a separate parallel one to support parallel
>>>>>>>>> functionalities (in possibility (b) all parallel-specific
>>>>>>>>> information
>>>>>>>>> is
>>>>>>>>> stored in a separate data to serial-specific data). We could
>>>>>>>>> design
>>>>>>>>> interface that would fit both possibilities, this is the reason
>>>>>>>>> why
>>>>>>>>> we
>>>>>>>>> included iPartMesh (section 3.4) level functions which are
>>>>>>>>> specific
>>>>>>>>> to
>>>>>>>>> parallel functionalities in mesh (and passing handle to serial
>>>>>>>>> data
>>>>>>>>> in
>>>>>>>>> these interface will not work for second possibility (b)). I do
>>>>>>>>> not
>>>>>>>>> know
>>>>>>>>> how much relevant is this.
>>>>>>>> I think it's very important to put the parallel data into our
>>>>>>>> existing
>>>>>>>> data models, or extend the data model if that can't be done (which
>>>>>>>> I
>>>>>>>> doubt, since I've already embedded this information in our current
>>>>>>>> data model).  This is important because it allows us to use current
>>>>>>>> tools to examine parallel data.  Note also that one may sometimes
>>>>>>>> be
>>>>>>>> examining parallel data in a serial environment (e.g. running a
>>>>>>>> partitioner in serial), so we shouldn't restrict accessing parallel
>>>>>>>> data to a parallel environment.
>>>>>>>>
>>>>>>>> - tim
>>>>>>>>
>>>>>>>>> Let me know if you have questions.
>>>>>>>>>
>>>>>>>>> Vitus, due to above questions I did not respond to your comments.
>>>>>>>>> I
>>>>>>>>> do not
>>>>>>>>> know if above questions and comments make things clear. If you
>>>>>>>>> still
>>>>>>>>> have
>>>>>>>>> anything specific please let us know.
>>>>>>>>>
>>>>>>>>> Karen, Is there a way to create an email list for this
>>>>>>>>> subcommittee
>>>>>>>>> as if
>>>>>>>>> we want to add (and remove) people (and it will keep record of all
>>>>>>>>> the
>>>>>>>>> messages). I have added Ting in this response as she is also
>>>>>>>>> working
>>>>>>>>> on
>>>>>>>>> this stuff from RPI side.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Onkar
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>    I'd like to re-iterate what I think should be the general
>>>>>>>>>> behavior of
>>>>>>>>>> functions in the parallel interface.  IMO, we should take
>>>>>>>>>> advantage
>>>>>>>>>> of
>>>>>>>>>> the simple yet powerful data model we already have for iMesh.
>>>>>>>>>> That
>>>>>>>>>> means using mesh sets for things which are obviously sets, and
>>>>>>>>>> using
>>>>>>>>>> the
>>>>>>>>>> functions which already exist for accessing data in those sets.
>>>>>>>>>> I
>>>>>>>>>> think
>>>>>>>>>> it does make sense in certain cases to make convenience functions
>>>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>> parallel interface for accessing e.g. all the sets which
>>>>>>>>>> represent
>>>>>>>>>> parts
>>>>>>>>>> of a partition, on a current processor or globally, but I think
>>>>>>>>>> we
>>>>>>>>>> should minimize these.
>>>>>>>>>>    Based on the general behavior described above, I think many of
>>>>>>>>>> the
>>>>>>>>>> functions proposed for global ids and parts/partitions already
>>>>>>>>>> exist
>>>>>>>>>> in
>>>>>>>>>> iMesh.  For example, iPart_getGlobalID, iPart_createGlobalID,
>>>>>>>>>> iProcParts_getPartIdArrOnProc, iProcParts iterator functions,
>>>>>>>>>> etc.
>>>>>>>>>>
>>>>>>>>>> - tim
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> ================================================================
>>>>>>>>>> "You will keep in perfect peace him whose mind is
>>>>>>>>>>   steadfast, because he trusts in you."               Isaiah 26:3
>>>>>>>>>>
>>>>>>>>>>              Tim Tautges            Argonne National Laboratory
>>>>>>>>>>          (tautges at mcs.anl.gov)      (telecommuting from
>>>>>>>>>> UW-Madison)
>>>>>>>>>>          phone: (608) 263-8485      1500 Engineering Dr.
>>>>>>>>>>            fax: (608) 263-4499      Madison, WI 53706
>>>>>>>>>>
>>>>>>>>>>
>>>>>> --
>>>>>> ================================================================
>>>>>> "You will keep in perfect peace him whose mind is
>>>>>>   steadfast, because he trusts in you."               Isaiah 26:3
>>>>>>
>>>>>>              Tim Tautges            Argonne National Laboratory
>>>>>>          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>>>>>>          phone: (608) 263-8485      1500 Engineering Dr.
>>>>>>            fax: (608) 263-4499      Madison, WI 53706
>>>>>>
>>>>>>
>>>>>
>>>> --
>>>> ================================================================
>>>> "You will keep in perfect peace him whose mind is
>>>>   steadfast, because he trusts in you."               Isaiah 26:3
>>>>
>>>>              Tim Tautges            Argonne National Laboratory
>>>>          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>>>>          phone: (608) 263-8485      1500 Engineering Dr.
>>>>            fax: (608) 263-4499      Madison, WI 53706
>>>>
>>>>
>>>
>>>
>> --
>> ================================================================
>> "You will keep in perfect peace him whose mind is
>>   steadfast, because he trusts in you."               Isaiah 26:3
>>
>>              Tim Tautges            Argonne National Laboratory
>>          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>>          phone: (608) 263-8485      1500 Engineering Dr.
>>            fax: (608) 263-4499      Madison, WI 53706
>>
>>
> 
> 
> 

-- 
================================================================
"You will keep in perfect peace him whose mind is
  steadfast, because he trusts in you."               Isaiah 26:3

             Tim Tautges            Argonne National Laboratory
         (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
         phone: (608) 263-8485      1500 Engineering Dr.
           fax: (608) 263-4499      Madison, WI 53706