itaps-parallel Re: Parallel Interface philosophical comments

[Onkar Sahni]

>> I see above comments at two levels, first is concerning a part (is it a
>> set?) and second is accounting inter-part boundaries like part edges,
>> part
>> verticies etc. (is it also a set?). In our implementation, both are not
>> entity-sets.
>>
>> Now, coming to the question why not use set parent/child relation
>> functions (or why not use sets for parts and inter-part boundaries);
>> there
>> are multiple reasons and I will try to give a simple but necessary one.
>> Consider two questions (that are elaborated in later part of email if
>> someone is interested):
>>
>> (a) For a giver part, which are the mesh entities residing on a
>> inter-part
>> boundary between say two parts (typically asked by applications like
>> flow
>> solver).
>> (b) For a specific mesh entity residing on a inter-part boundary, what
>> are
>> its residence parts (typically asked by applications like mesh
>> modification software).
>>
>> It basically boils down to classification and reverse-classification (as
>> we discussed in our last boot camp). If we only have
>> reverse-classification question (b) becomes expensive and inconvenient
>> to
>> answer (for problems of practical interest, that would eventually be
>> petascale).
>>
>
> Yes.  To re-iterate, we want to allow for both and require neither.
> However, in the case where something is done, I think we should use
> existing functions to do it.  That doesn't mean you have to implement it
> that way underneath.
>
> Instead of classification and reverse classification terms, I wish we'd
> use entity-based and set-based; the classification term has a previous
> usage implying geometry, which is confusing, at least to me.  I know
> saying set-based may imply to others that entity sets are used to
> implement it, though; maybe we need to find another, neutral term
> (collection?).  Carl, I don't want to hear anything about Fred or George.
>
> So, for entity-based things, I think we should use existing tag
> functions, either with a conventional tag name, or an
> application-specified tag name.  For set-based things, we should use the
> existing set functions.  This follows well-known principles in API
> design (for example, see http://lcsd05.cs.tamu.edu/slides/keynote.pdf
> and http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=488).

Let me try to explain question (b) in a different context. Consider
function: iMesh_getEntTopo(). This function provide the topology of a
particular mesh entity. Is this API good (to a user)? Yes! easy to learn,
easy to use, to the point, frequently used etc. I guess this is not
difficult to tell but let me add on to why it is really "good" by giving a
contrasting example.

Say I apply tag-based approach (for now, entity-to-sets) where I create a
set for each entity-topology. Now a user wants to know same information
for a specific mesh entity, what will a user do. Not difficult to guess,
under this approach first get the set from entity (using either a
conventional tag name, or an application-specified tag name) and then use
another API (which is not really existing yet, not that I know of) to ask
whats the property of a set (i.e., ask for entity-topology). One can
argue, why not create tags/tag names based on entity-topology, so user
doesn't have to ask for any property of set, then what does user do in
this case, I guess check for all tags based on every entity-topology and
if finds one (attached set) then user figures out from that tag/tag name
whats the topology of the entity.

Now lets come back to original question (b), where user wants to know
residence parts for a specific entity. Guess what, unlike entity-topology
the combinations of residence-parts is unknown before-hand, and also its
much greater in number for real problems (involving 1000 or 100000 parts).

So the point is do we want to provide user with "good" (and
single/complete) APIs (for relevant things, like iPart_getEntResParts())
or we just want to tell users that use existing APIs (that are very useful
for the purposes they are designed for) to figure out a (cumbersome) way
to do petascale MIMD computing.

Moreover, we do not have a user-base yet (using iPart interface) so in a
way we can be careful in design of parallel functionalities.

In any case, one more point I wanted to make is that in our submission we
have not proposed changes in existing APIs (as we agree that there can be
huge cost and may be loss in changing existing APIs).

In this email I have limited my discussion on above issues.

- Onkar

>> For (re-)partitioning of meshes, my understanding is that we would call
>> partitioner lib. (like Zoltan) that will provide us destination part ids
>> for part objects (which are typically regions for mesh on 3D manifold).
>> Then we would perform mesh migration to create/fill/form the parts. Now,
>> when a partitioner lib. (Zoltan) is called I am assuming we already have
>> the mesh to start with (in case its initial partitioning step then there
>> is only one part to begin with or in case if its re-partitioning step
>> then
>> there could be on one or more parts in the start). After call to
>> partitioned lib., i.e., during mesh migration, mesh entities will be
>> created (and some deleted) from part(s) as required. I agree with Carl's
>> point 6 on mesh modification in his submission, which is somewhat
>> related
>> to this discussion.
>
> We also want to allow the construction of a partitioning tool above the
> level of iMesh, such as the itaps-zoltan tool that Vitus has written.
> This tools is an iMesh application which grabs the mesh graph
> information, calls Zoltan to generate the partition, and embeds that
> partition into iMesh.  A couple important principles emerge when
> considering that example, in reference to your comments:
>
> a) You might want to do a static partition, in parallel, starting with
> some other partitioning (e.g. material definitions or geometric topology
> groupings).  This is different than dynamic partitioning, where an
> existing parallel application wants to re-partition and migrate its
> data.  In this case, the partitioner still needs to write the partition
> onto the mesh, preferably as a tool sitting on top of iMesh.
>
> b) You might want to write multiple partitions onto the same mesh, or
> consider different collections of coverings on a given mesh.  For
> example, you might not be able to load the mesh on a single processor,
> so you use a coarse partition/covering to load the mesh in parallel,
> then call a parallel static partitioner.
>
> To handle these cases, you're going to need to extend the current
> proposed spec to handle multiple partitions.  The current spec for sets
> and tags, though, will handle this just fine.
>
>>
>> Open question is how much control do we want to provide the
>> user/application during (re-)partitioning step (say, do we want the
>> user/application to do all the somewhat tedious job of mesh migration
>> and/or we allow an input of destination part ids and we internally
>> perform
>> mesh migration), again depends who the user/application is. For cases
>> with
>> multi-parts per processor/process, we can go into further details when
>> necessary.
>
> I think we need to separate the notion of putting a mesh entity in a
> different part and migrating mesh to a different processor (I think the
> latter could be phrased as synching the parallel mesh layout to a
> collection of parts).
>
>>
>>> I was referring to your statement below, "For existing functions we can
>>> support compatibility with part-handles but main concern is for new
>>> ones
>>> (which doesn't do existing/same things).".  Maybe by "new ones" you
>>> meant the functions for this parallel interface?
>>>
>>
>> Yes, by "new ones" I meant functions for this parallel interface (which
>> we
>> are trying to define).
>>
>> ---------------------------------
>> Questions on classification and reverse-classification elaborated (if
>> someone is interested):
>>
>> (a) Consider a case where user (doing flow computations) wants to create
>> communication trace between two parts for a flow solver (basically to
>> see
>> which dofs need to talk to each other between two parts), so user asks
>> for
>> mesh entities (which can be translated to dofs) that reside on both
>> parts
>> at inter-part boundaries (for simplicity consider that there is no
>> ghosting involved). Basically we can provide a mechanism based on sets
>> where the set contains all these entities (which we referred as
>> reverse-classification in boot camp).
>>
>> (b) Now, consider a different task where user (who is performing mesh
>> modifications) wants to know residence parts for a particular mesh
>> entity
>> residing on a inter-part boundary. Considering there is only
>> reverse-classification available (through sets), the way I think user
>> can
>> get this information is to first figure out the set which contains this
>> mesh entity and then query information about that set (which contains
>> the
>> mesh entity). Now, in real cases involving unstructured meshes on say
>> 100
>> or 1000 parts, even by very modest estimates there will be say 50 sets
>> for
>> all inter-part boundaries it would be quite expensive and inconvenient
>> for
>> user to figure this out (now number of sets providing all inter-part
>> boundaries depends on how things are defined but in any case the
>> complexity involved will be significant if followed through
>> reverse-classification approach).
>
> Absolutely.  This is analogous to iRel, where you want the option of
> relating both entities and sets to entities or sets, for efficiency.
> It's a memory vs. speed tradeoff, and the application has to be able to
> choose.  This is another example of where our existing tag and set
> functionality would help us, since we can tag both entities and sets.
>
> - tim
>
>> ---------------------------------
>>
>> - Onkar