itaps-parallel Re: Parallel Interface philosophical comments
Onkar Sahni
osahni at scorec.rpi.edu
Thu Oct 25 17:44:07 CDT 2007
>> For clarification, by operations on entity sets I meant set-specific
>> operations/functions, like iMesh_addEntToSet, iMesh_rmvEntFromSet,
>> iMesh_subtract, iMesh_intersect, iMesh_unite etc. My understanding, from
>> the last boot camp, is that these functions will not be enabled to
>> accept
>> part-handles.
>
> I'm fine with that for set booleans, but for remove/add, I'm not sure.
> How will a partitioner go about building partition sets? And, if you
> have things like part edges and part vertices (fig 3 in your pmodel1.3
> document), why not use the set parent/child relation functions to
> retrieve those relations instead of having new functions for that?
>
I see above comments at two levels, first is concerning a part (is it a
set?) and second is accounting inter-part boundaries like part edges, part
verticies etc. (is it also a set?). In our implementation, both are not
entity-sets.
Now, coming to the question why not use set parent/child relation
functions (or why not use sets for parts and inter-part boundaries); there
are multiple reasons and I will try to give a simple but necessary one.
Consider two questions (that are elaborated in later part of email if
someone is interested):
(a) For a giver part, which are the mesh entities residing on a inter-part
boundary between say two parts (typically asked by applications like flow
solver).
(b) For a specific mesh entity residing on a inter-part boundary, what are
its residence parts (typically asked by applications like mesh
modification software).
It basically boils down to classification and reverse-classification (as
we discussed in our last boot camp). If we only have
reverse-classification question (b) becomes expensive and inconvenient to
answer (for problems of practical interest, that would eventually be
petascale).
For (re-)partitioning of meshes, my understanding is that we would call
partitioner lib. (like Zoltan) that will provide us destination part ids
for part objects (which are typically regions for mesh on 3D manifold).
Then we would perform mesh migration to create/fill/form the parts. Now,
when a partitioner lib. (Zoltan) is called I am assuming we already have
the mesh to start with (in case its initial partitioning step then there
is only one part to begin with or in case if its re-partitioning step then
there could be on one or more parts in the start). After call to
partitioned lib., i.e., during mesh migration, mesh entities will be
created (and some deleted) from part(s) as required. I agree with Carl's
point 6 on mesh modification in his submission, which is somewhat related
to this discussion.
Open question is how much control do we want to provide the
user/application during (re-)partitioning step (say, do we want the
user/application to do all the somewhat tedious job of mesh migration
and/or we allow an input of destination part ids and we internally perform
mesh migration), again depends who the user/application is. For cases with
multi-parts per processor/process, we can go into further details when
necessary.
> I was referring to your statement below, "For existing functions we can
> support compatibility with part-handles but main concern is for new ones
> (which doesn't do existing/same things).". Maybe by "new ones" you
> meant the functions for this parallel interface?
>
Yes, by "new ones" I meant functions for this parallel interface (which we
are trying to define).
---------------------------------
Questions on classification and reverse-classification elaborated (if
someone is interested):
(a) Consider a case where user (doing flow computations) wants to create
communication trace between two parts for a flow solver (basically to see
which dofs need to talk to each other between two parts), so user asks for
mesh entities (which can be translated to dofs) that reside on both parts
at inter-part boundaries (for simplicity consider that there is no
ghosting involved). Basically we can provide a mechanism based on sets
where the set contains all these entities (which we referred as
reverse-classification in boot camp).
(b) Now, consider a different task where user (who is performing mesh
modifications) wants to know residence parts for a particular mesh entity
residing on a inter-part boundary. Considering there is only
reverse-classification available (through sets), the way I think user can
get this information is to first figure out the set which contains this
mesh entity and then query information about that set (which contains the
mesh entity). Now, in real cases involving unstructured meshes on say 100
or 1000 parts, even by very modest estimates there will be say 50 sets for
all inter-part boundaries it would be quite expensive and inconvenient for
user to figure this out (now number of sets providing all inter-part
boundaries depends on how things are defined but in any case the
complexity involved will be significant if followed through
reverse-classification approach).
---------------------------------
- Onkar
>
>>>> (c) The new interface functions, we need to define for parts (to
>>>> support
>>>> parallel requirements/functionalities), will take part-handles as
>>>> input
>>>> and not any set-handles (if some implementation wants to use
>>>> entity-sets
>>>> for parts they can have part-handle and entity-set-handle as one and
>>>> the
>>>> same).
>>> I agree that it will have to be an error if an application passes in a
>>> set handle representing a set and not a part to one of the functions
>>> expecting a part handle. It'll have to be up to the implementation to
>>> decide how to do that (using casting, looking for the tag designating
>>> part-type sets, etc.)
>>
>> I guess I am not sure what you mean by set-handle not representing a
>> part
>> (or part-type sets). In our implementation there will be no set-handle
>> representing a part (or part-type sets) rather there will be
>> part-handle.
>> I think there is a strong distinction between these two things, i.e.,
>> set/set-handle and part/part-handle (as a part need not necessarily be a
>> set in an implementation so why call part-type set or set-handle
>> representing a part).
>>
>
> Sorry, I should have said "part handle" or "set handle" instead of
> part-type set.
>
> - tim
>
>> - Onkar
>>
>>> - tim
>>>
>>>>> iRel is for relating data between different interfaces. As I
>>>>> understand
>>>>> it, we are building parallel functionality as part of iMesh.
>>>>> Therefore,
>>>>> we shouldn't need to use iRel for this.
>>>>>
>>>>>> There are a number of groups that use classification as the primary
>>>>>> mode
>>>>>> of interaction for doing things similar to how the RPI group does.
>>>>>> Just
>>>>>> as we would not want you to have redo all your parallel work to do
>>>>>> things strictly from a classification perspective, we do not expect
>>>>>> that
>>>>>> the ITAPS parallel interface will require RPI to re-do all its
>>>>>> parallel
>>>>>> adaptive work (years worth of work dating back to the mid-90's when
>>>>>> we
>>>>>> were the first to do it) that is being used for SciDAC applications
>>>>>> to
>>>>>> fit with a specific one way of doing things.
>>>>>>
>>>>>> By the way, some feel the classification is a simple and powerful
>>>>>> way
>>>>>> to
>>>>>> do things and mesh sets are actually substantial extra overhead to a
>>>>>> well defined model of using classification. In my experience, I
>>>>>> would
>>>>>> love to have both classification and reverse classification at all
>>>>>> times, since depending on the question being asked, one is a faster
>>>>>> way
>>>>>> to answer it. However, due to the extra data and headache of keeping
>>>>>> things correctly set, most I know tend to focus on one and use the
>>>>>> other
>>>>>> temporary manner when its advantageous.
>>>>>>
>>>>> I'm not advocating that we do things only with reverse
>>>>> classification,
>>>>> I
>>>>> agree we need to be able to go both ways. I'm just arguing that if
>>>>> we
>>>>> do pass part handles through the interface, then we operate on those
>>>>> part handles using existing functions, rather than make new ones that
>>>>> do
>>>>> virtually the same thing.
>>>> For existing functions we can support compatibility with part-handles
>>>> but
>>>> main concern is for new ones (which doesn't do existing/same things).
>>>>
>>>> - Onkar
>>>>
>>>>
>>>>> - tim
>>>>>
>>>>>> Mark
>>>>>>
>>>>>> Tim Tautges wrote:
>>>>>>> Onkar Sahni wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Here are some of my questions and comments on ITAPS parallel
>>>>>>>> interface:
>>>>>>>>
>>>>>>>> 1) Is there a difference between "instance" and "handle" in ITAPS
>>>>>>>> terminology (or are there any definition for these)? (I may have
>>>>>>>> missed it
>>>>>>>> if this was resolved/defined in previous ITAPS interfaces like
>>>>>>>> iMesh).
>>>>>>>>
>>>>>>> We use the generic term "handle" as an identifier for entities,
>>>>>>> entity
>>>>>>> sets, the interface itself, and tags. In our C headers, these are
>>>>>>> typedef'd to void*. There are also iterators, which we haven't
>>>>>>> really
>>>>>>> called handles, but could be referred to that way.
>>>>>>>
>>>>>>>> 2) For situations where we want to support multi-parts per
>>>>>>>> processor/process:
>>>>>>>> (a) How do we want to distinguish between processor/process level
>>>>>>>> functionalities against part level ones (keeping user- and
>>>>>>>> implementation-convenience and efficiency in mind)? Ones which
>>>>>>>> would
>>>>>>>> fall
>>>>>>>> under processor level are, like list of parts on processor,
>>>>>>>> MPI_Rank
>>>>>>>> etc.,
>>>>>>>> whereas ones which would fall under part level are, like
>>>>>>>> neighboring
>>>>>>>> parts
>>>>>>>> (not procs.) etc. Further, there could be ones where users wants
>>>>>>>> to
>>>>>>>> get/iterate mesh entities (and get adjacencies) at a processor
>>>>>>>> and/or
>>>>>>>> part
>>>>>>>> level.
>>>>>>> Processor-level queries are on the iMesh instance, part-level
>>>>>>> queries
>>>>>>> are on the entity set representing the part. Our iterator
>>>>>>> functions
>>>>>>> operate on both (with processor-level queries simply using the root
>>>>>>> set).
>>>>>>>
>>>>>>>> (b) Assuming a part-handle is supported, how can user/application
>>>>>>>> get
>>>>>>>> access to it (say on a processor where part resides)? Basically
>>>>>>>> will
>>>>>>>> there
>>>>>>>> be a way to directly access the part-handle (like part iterator at
>>>>>>>> iProcParts level) or user has to access it through tags/tag names,
>>>>>>>> in
>>>>>>>> which case user has to figure out what is the tag/tag name for a
>>>>>>>> particular part (and who defines these tags/tag names).
>>>>>>> I advocate having both: a conventional tag given to parts, so that
>>>>>>> you
>>>>>>> can find sets that are parts by looking for sets with that tag; and
>>>>>>> also having a convenience function (as part of the parallel
>>>>>>> extensions
>>>>>>> to iMesh) which returns all the parts in the given instance (i.e.
>>>>>>> on
>>>>>>> the processor). Note an important capability here for this
>>>>>>> function
>>>>>>> is to pass in an optional tag name/handle indicating the partition
>>>>>>> you're asking about. That allows multiple partitions to co-exist,
>>>>>>> which is important for some applications.
>>>>>>>
>>>>>>>> 3) How general we want to be in interface for different data
>>>>>>>> models/implementations (as in future we may want to
>>>>>>>> incorporate/fit
>>>>>>>> other
>>>>>>>> implementations), especially between serial and parallel data. Two
>>>>>>>> possibilities for data models I can think of are: (a) one (serial)
>>>>>>>> data
>>>>>>>> model is enhanced to store parallel information and (b) serial one
>>>>>>>> is
>>>>>>>> supplemented by a separate parallel one to support parallel
>>>>>>>> functionalities (in possibility (b) all parallel-specific
>>>>>>>> information
>>>>>>>> is
>>>>>>>> stored in a separate data to serial-specific data). We could
>>>>>>>> design
>>>>>>>> interface that would fit both possibilities, this is the reason
>>>>>>>> why
>>>>>>>> we
>>>>>>>> included iPartMesh (section 3.4) level functions which are
>>>>>>>> specific
>>>>>>>> to
>>>>>>>> parallel functionalities in mesh (and passing handle to serial
>>>>>>>> data
>>>>>>>> in
>>>>>>>> these interface will not work for second possibility (b)). I do
>>>>>>>> not
>>>>>>>> know
>>>>>>>> how much relevant is this.
>>>>>>> I think it's very important to put the parallel data into our
>>>>>>> existing
>>>>>>> data models, or extend the data model if that can't be done (which
>>>>>>> I
>>>>>>> doubt, since I've already embedded this information in our current
>>>>>>> data model). This is important because it allows us to use current
>>>>>>> tools to examine parallel data. Note also that one may sometimes
>>>>>>> be
>>>>>>> examining parallel data in a serial environment (e.g. running a
>>>>>>> partitioner in serial), so we shouldn't restrict accessing parallel
>>>>>>> data to a parallel environment.
>>>>>>>
>>>>>>> - tim
>>>>>>>
>>>>>>>> Let me know if you have questions.
>>>>>>>>
>>>>>>>> Vitus, due to above questions I did not respond to your comments.
>>>>>>>> I
>>>>>>>> do not
>>>>>>>> know if above questions and comments make things clear. If you
>>>>>>>> still
>>>>>>>> have
>>>>>>>> anything specific please let us know.
>>>>>>>>
>>>>>>>> Karen, Is there a way to create an email list for this
>>>>>>>> subcommittee
>>>>>>>> as if
>>>>>>>> we want to add (and remove) people (and it will keep record of all
>>>>>>>> the
>>>>>>>> messages). I have added Ting in this response as she is also
>>>>>>>> working
>>>>>>>> on
>>>>>>>> this stuff from RPI side.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Onkar
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>> I'd like to re-iterate what I think should be the general
>>>>>>>>> behavior of
>>>>>>>>> functions in the parallel interface. IMO, we should take
>>>>>>>>> advantage
>>>>>>>>> of
>>>>>>>>> the simple yet powerful data model we already have for iMesh.
>>>>>>>>> That
>>>>>>>>> means using mesh sets for things which are obviously sets, and
>>>>>>>>> using
>>>>>>>>> the
>>>>>>>>> functions which already exist for accessing data in those sets.
>>>>>>>>> I
>>>>>>>>> think
>>>>>>>>> it does make sense in certain cases to make convenience functions
>>>>>>>>> in
>>>>>>>>> the
>>>>>>>>> parallel interface for accessing e.g. all the sets which
>>>>>>>>> represent
>>>>>>>>> parts
>>>>>>>>> of a partition, on a current processor or globally, but I think
>>>>>>>>> we
>>>>>>>>> should minimize these.
>>>>>>>>> Based on the general behavior described above, I think many of
>>>>>>>>> the
>>>>>>>>> functions proposed for global ids and parts/partitions already
>>>>>>>>> exist
>>>>>>>>> in
>>>>>>>>> iMesh. For example, iPart_getGlobalID, iPart_createGlobalID,
>>>>>>>>> iProcParts_getPartIdArrOnProc, iProcParts iterator functions,
>>>>>>>>> etc.
>>>>>>>>>
>>>>>>>>> - tim
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> ================================================================
>>>>>>>>> "You will keep in perfect peace him whose mind is
>>>>>>>>> steadfast, because he trusts in you." Isaiah 26:3
>>>>>>>>>
>>>>>>>>> Tim Tautges Argonne National Laboratory
>>>>>>>>> (tautges at mcs.anl.gov) (telecommuting from
>>>>>>>>> UW-Madison)
>>>>>>>>> phone: (608) 263-8485 1500 Engineering Dr.
>>>>>>>>> fax: (608) 263-4499 Madison, WI 53706
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>> --
>>>>> ================================================================
>>>>> "You will keep in perfect peace him whose mind is
>>>>> steadfast, because he trusts in you." Isaiah 26:3
>>>>>
>>>>> Tim Tautges Argonne National Laboratory
>>>>> (tautges at mcs.anl.gov) (telecommuting from UW-Madison)
>>>>> phone: (608) 263-8485 1500 Engineering Dr.
>>>>> fax: (608) 263-4499 Madison, WI 53706
>>>>>
>>>>>
>>>>
>>>>
>>> --
>>> ================================================================
>>> "You will keep in perfect peace him whose mind is
>>> steadfast, because he trusts in you." Isaiah 26:3
>>>
>>> Tim Tautges Argonne National Laboratory
>>> (tautges at mcs.anl.gov) (telecommuting from UW-Madison)
>>> phone: (608) 263-8485 1500 Engineering Dr.
>>> fax: (608) 263-4499 Madison, WI 53706
>>>
>>>
>>
>>
>>
>
> --
> ================================================================
> "You will keep in perfect peace him whose mind is
> steadfast, because he trusts in you." Isaiah 26:3
>
> Tim Tautges Argonne National Laboratory
> (tautges at mcs.anl.gov) (telecommuting from UW-Madison)
> phone: (608) 263-8485 1500 Engineering Dr.
> fax: (608) 263-4499 Madison, WI 53706
>
>
More information about the itaps-parallel
mailing list