itaps-parallel Comments on parallel stuff so far.

[Onkar Sahni]

Overall I agree with Carl's comments, mainly the idea of considering
things at different levels, i.e., partition, processor and part levels. I
have some general comments first and then specific ones.

Here are general comments:

(a) I see at each level there are two sections: "Data" and "New
Functionality". As long as the "New Functionality" section (through
certain queries like getTotNumOfParts etc.) is satisfied we do not want
any strict requirement on "Data" section. For example, some
implementation(s) may not want to explicitly store "IDs of all parts" at
partition level but will satisfy/fulfill things in "New Functionality"
section.

(b) We may want to consider which levels interact with each other or which
functionalities lies at what level. For example, do we want to keep
addPartOnProc, rmvPartOnProc (not rmvProcOnPart, Carl I think this was a
typo in your email) at partition level or processor level. Other way to
put it is, can partition and part level interact directly with each other
or we allow interactions between part-processor levels and
processor-partition levels (as processor level sits in between of part and
partition level). A question to Carl, is there a reason to have both
addPartOnProc/rmvPartOnProc and createPart/destroyPart?

(c) At part level, for entities (owned or non-owned) on inter-part
boundaries we would need handles and part Ids of its copies/images on
other parts (even if we have multi-parts per processor).

(d) One thing we may want to be clear in future is that processor (as used
in our discussions up till now) really means process (an MPI task) and not
physical hardware as one could load multiple processes (many MPI tasks) on
a single physical processor (if possible).

Here are some specific comments:

(a) Some interface functions might be redundant but are provided for
convenience (getNbhgPartIds and getNbhgProcRanks). If we think it is not
important/useful to provide then we can ignore them. Other example is part
iterators, which may be useful for many parts per process/processor (like
40 for nodes with 40 cores).

(b) getPartMesh: If a mesh instance on a process/processor knows which
partition it belongs to then a part should know which mesh instance it
belongs to.

(c) I think it would be good to combine isEntOnPartBdry, isEntOwner and
isEntGhost into one.

(d) I like the idea of
iPart_sendEnt[Arr]ToParts/iPart_receiveEnt[Arr]OnParts. My questions is
whether we want to hide the pair of calls inside a single interface like
iPart_sendReceiveEnt[Arr] (or iPart_migrateEnt[Arr], if migrate is a
neutral enough term).

- Onkar

> Hello, all.
>
> For a while there, things were moving too fast for me to keep up. :-)
> Now that I've read through all the email, I've got some comments, which
> I'm sure comes as a shock to all of you. :-)
>
> First, some comments on the long thread about (essentially) using
> existing set functionality versus not:
>
> Part of the problem may be that we currently have no clearly-defined
> top-level partition.  As a result, we don't have a way to create a new
> part (and keep track of it).  We also don't have a way to segregate two
> separate partitionings.  Some of Tim's comments (for example, about the
> use of a tag convention to identify which set handles are parts, and
> presumably to use the value of that tag to identify which partition they
> belong to) may stem from the need to carry multiple partitions at once,
> for which there's currently no proposal about functionality.
>
> (Question as an aside for Tim:  Are you thinking about different
> partitionings of the same mesh, or different meshes on the same domain
> with different partitionings?  If the former, are the partitionings
> nested, or not necessarily?)
>
> We should at least consider having a "partition" object that holds all
> the top-level data, including handles for all the parts, part-processor
> mapping, maybe an MPI communicator for the partition, etc.  Then we can
> create/destroy parts (including registering them with the partition;
> obviously creating the partition with some number of parts is a great
> piece of common shorthand), assign them to processors, report on the
> mapping of parts to processors, etc, which is functionality that may be
> (as I read it) only partially proposed at this point.  I recognize that
> some of these functions may turn out to be trivial, but conceptually
> they're still worth considering because it's functionality that pretty
> clearly needs to be there.  Obviously some of this functionality is
> already proposed in various pieces that are currently under discussion.
>
> This level also seems to be the logical place for global calls, like
> those Tim proposes, as well as some of those in the RPI document (like
> getTotNumOfParts, getNumOfPartsArr, etc).  An advantage to this is that
> we can then use a partition handle of some sort in calls like
> getGlobalPartOnProc (one of numerous examples) that require the ability
> to distinguish between multiple partitions when multiple partitions exist.
>
> Also, we need to be able to identify which partition a part belongs to.
>   This can be a function or a tag convention; I'm thinking of it just as
> functionality at this point; whether it eventually is a function or a
> conventional tag or something else is a question for a bit later.
>
> Overall, what I'm proposing looks more or less like the following:
>
> Partition level:
>
>      Data:
>      . IDs of all parts (every part belongs to a partition, like every
>        serial set belongs to a root set)
>      . mapping of parts to processors
>      . the existence of a list of processors implies an MPI communicator,
>        presumably; whether/how to expose this communicator to apps?
>      New Functionality:
>      . getTotNumOfParts
>      . getNumOfPartsArr
>      . global<->local part ID mapping and part<->processor mapping,
>        supplementing or superceding what's currently proposed
>        (getPartIDArrOnProc, isPartOnProc, getGlobalPartOnProc,
>        getLocalPartOnProc)
>      . addPartOnProc, rmvProcOnPart
>      . createPart, destroyPart
>      . getNumEntsPar, getNumEntSetsPar, getNumEntsProcPar,
>        getNumEntSetsProcPar
>      . getTagsPar, getTagsProcPar, get by tag
>      . global reduce on tags
>      . get/setGlobalIDSize
>
> Processor level:
>
>      Data:
>      . IDs of all parts on that processor
>      . iMesh instance
>      New Functionality:
>      . getNumOfPartsOnProc
>      . getPartsOnProc (all part handles)
>      . part iterator (could be replaced by mapping local ID to part
>        handle)
>      . local ID <-> (local) part handle
>      . iMesh handle
>      . what partition do I belong to?
>
> Part level:
>
>      Data:
>      . owned entities
>      . copies of non-owned interpart bdry entities
>      New Functionality:
>      . getNghbPartIds (if this returns global part IDs (proc rank + local
>        ID), then get getNghbProcRanks is redundant)
>      . part bdry iterators (both single and array versions)
>      . part bdry size info
>      . getOwnerOfEnt, getCopiesOfEnt, getCopyOfEnt, getNumOfCopiesOfEnt
>      . entity categorization within part
>      . addCopyOfEnt, rmvCopyOfEnt
>      . disambiguate ownership of shared entity
>      . what partition do I belong to?
>      . get/set/createGlobalID[Arr]
>      . getEntHandle[Arr] from global ID
>
>
> ======================================================================
>
> On the question of how to add entities to parts, I think we need to
> examine the use cases for adding entities to parts.  I can think of
> several scenarios:
>
>    1.  A new entity has been created by a modify operation and needs to
> be associated with a part.
>    2.  An entity is being migrated from one part to another (small
> one-to-one migration due to change of ownership; data is almost
> certainly moving from one part to another).
>    3.  A copy is being created (depending on the implementation, some
> data may have to copied from one part to another).
>    4.  A new partition has been created and parts must be populated
> (massive all-to-all migration; vast amounts of data flying everywhere).
>
> That list probably isn't exhaustive, but it'll do for now.  Of those
> four, I see one, possibly two, that -don't- involve parallel data
> transfer.  Also all (potentially) will affect not only ownership of the
> primary entities partitioned, but also the part boundaries.  At the very
> least, we need to be very clear that we're doing things that involve
> parallel communication here.   One possible way to handle this:
>
> o We could handle scenario 1 cleanly with set syntax, with the
> -specific- caveat clearly documented that this function is for use only
> for local entities.  This is enforceable by requiring the entity args to
> be handles (necessarily local to the processor, though not the part) and
> mandating an error return when trying a back-door migration between
> parts on a single processor.  This is certainly possible, but is it
> common enough and unambiguous enough to justify doing it, especially in
> the face of a need for a different mechanism for other scenarios?
>
> o As I just hinted at, I don't think the current set calls are
> appropriate for Scenarios 2-4 (and other migration scenarios), because
> the syntax then hides that there's parallel communication going on (as
> well as the issue of needing to be able to grab a communicator,
> possibly).  Among other viable approaches, the migration scenarios could
> be handled by an explicit sendEntToPart/receiveEnts call pair, which I
> envision using syntax that is qualitatively like this:
>
> iPart_sendEntArrToParts(source_part_handle, ent_handle_array,
> target_part_ID_array, MIGRATE);
> iPart_sendEntArrToParts(source_part_handle, ent_handle_array,
> target_part_ID_array, COPY);
> iPart_receiveEnts;
>
> Yes, I can envision potentially sending ghost data using this same
> approach.  And yes, the actual data packaged up and sent from part to
> part is going to need to be implementation dependent (and possibly
> application specifiable...).  And yes again,
> sendEntArrToParts(...MIGRATE) should delete local data; when is a good
> question... No, I don't think single-entity versions of these are likely
> to be a good idea for scalability reasons.
>
> The last call (receiveEnts) would block waiting for data; a broadcast up
> front is probably needed to make sure every processor knows what
> messages it has to wait for.  Once all the data is received, then the
> receiveEnts call can also do whatever implementation-dependent things
> are needed to make sure the interior/bdry/ghost info has been properly
> updated internally.
>
> I like the pair of calls not only because it gives a natural place to
> update things like part bdry info, but also because it allows at least
> some latency hiding, which an immediate, one-call migrate functionality
> doesn't.
>
> ================================================================
>
> And now that I've no doubt said something that everyone will agree with
> and something that everyone will disagree with, I'll move on to some
> specific comments. :-)
>
> RPI (Parts, etc)
>
> I had some comments about iProcParts, iPart, iPartMesh that are now
> covered by the discussion above.
>
> part iterator: Natural analog to entities, BUT do we really need this,
> given that the array of part handles for a given processor is likely to
> be small enough that we won't mind just grabbing the whole array and
> iterating outside the interface?
>
> getPartMesh: Given that we can call all applicable iMesh functions with
> a part handle, do you see a role for this other than mesh modification?
>
> [Comments on part<->processor mapping subsumed above.]
>
> isEntOnPartBdry, isEntOwner, isEntGhost: I think we should combine these
> into a single call with an enumerated "return" value.  Simpler now, and
> allows for future expansion in the list of properties we care about, if
> needed.
>
> Also, need a way to set/negotiate ownership of shared entities on part
> boundaries.
>
> Tim's stuff:
>
> getEntByTypeAndTagPar, getEntSetByTagPar: We don't have this
> functionality in serial.  Maybe we should, but we currently don't.
>
> getTag*Operate: Again, we haven't got this in serial.  Does the
> existence of such operations imply that we expect to implement fields as
> tags? (Because that wasn't what I was assuming about field
> implementations at all, personally...)  Note that I'm not opposed to
> this sort of global reduction operation, I just wonder whether it'll see
> use outside of field-like situations.  If not, then it should be in
> parallel fields, not parallel mesh, and usage for
> fields-implemented-as-tags should be handled there.
>
> Global IDs:
>
> One question here.  As I understand it, the intention that
> getEntHandle[Arr] will return a local handle (owned, copied, or ghosted)
> for the global ID if possible, and otherwise give an error.  Is that
> correct?
>
> I now return you to your regularly scheduled afternoon.
>
> Carl
>
>
> --
> ------------------------------------------------------------------------
> Dr. Carl Ollivier-Gooch, P.Eng.                   Voice: +1-604-822-1854
> Associate Professor                                 Fax: +1-604-822-2403
> Department of Mechanical Engineering             email: cfog at mech.ubc.ca
> University of British Columbia              http://www.mech.ubc.ca/~cfog
> Vancouver, BC  V6T 1Z4                  http://tetra.mech.ubc.ca/ANSLab/
> ------------------------------------------------------------------------
>
>