itaps-parallel Part handles

[Tim Tautges]

Hi all,
   I realize now why I missed this discussion in March, it was in the
middle of a 3 week/3 trip stretch for me, and I got overwhelmed and
didn't follow the discussion closely.

Anyway, this email is an attempt to summarize my objection to embedding
the proc rank in the part handle.  As background, you should know that I
almost went this way in MOAB (to the point where it's still implemented
in MOAB) but ultimately decided not to, for many of the same reasons I
note below.

And, sorry if this is dumb and/or offensive, but I still don't see a 
logical, non-pejorative statement of the problem whose only, or most 
logical, solution is the two primary requirements causing me such 
heartburn (part handles must be unique, and processor must be derivable 
from part handle).  And that's after reading the March email thread at 
least three times, and thinking about this for at least 4 hours over the 
last couple of days.

The remainder of this email consists of 3 parts:
Part I: my objections
Part II: questions about the requirements Karen sent out Wed afternoon
Part III: questions/comments arising while looking through current
interface draft


Part I

1. If we are required to embed the proc rank in the handle, then either 
I cannot use entity sets to represent parts, or I have to embed the proc 
in all my set handles, which allows at most 2**(32-n)-1 other sets 
available on any processor, where n is the log of the number of 
processors; if we need to support millions of processors, n >= 20, 
leaving me only 4k sets.  Putting that kind of limitation on the number 
of sets will not be acceptable for some important applications of sets, 
e.g. geometric topology sets, sets to represent kdtree decomposition of 
the local mesh (for point location), or OBB tree decomposition of 
surface triangulations (for ray firing).  Saying we must not implement 
parts as sets is at least as bad as saying we must implement parts as 
sets; furthermore, it prevents the dual-use of a given set as a part 
also (something which I already do, and which is quite useful).

2. Deriving the proc from a part handle will still need to be done
through a function in iMeshP, to avoid forcing applications to do
bit-wise operations on the handles (e.g. Fortran 77 doesn't have bitwise
operations).  If we need to do this interpretation in a function anyway, 
why not also let the implementation decide how to track the part/proc 
assignment?

3. If you allow applications to directly find the proc from the part 
handle (without going through an iMesh function), then besides 
specifying embedding the proc in the handle, you'll also
need to specify exactly how that's done, e.g. which bits hold the rank,
and whether that number of bits can change depending on the max # procs
you'd like to support with that mesh.  Are we confident we can get a
consensus on that?

4. If the application is telling Zoltan about the parts, and a part can 
reside on only one processor, then doesn't that mean that Zoltan finds 
out where the parts reside at the same time?  If messages are sent 
containing the part handle, is it so much trouble to add one extra 
integer denoting the proc where that part resides?  If it's a point to 
point message, you won't even have to do that, since you know where the 
message came from (for MPI, anyway).

Part II

Also, some comments on the proposed requirements and justification of them:

   + Part handles are globally unique (so that remote part handles make
sense and Zoltan doesn't go insane);
- remote part handles won't exist on a processor unless they were 
communicated there somehow; in that case the implementation will be able 
to store the proc # with (not in) the handle if it so chooses
- didn't I hear you say, Karen, that Zoltan computes its own 0..n-1
numbering of parts anyway?  Or, will you eliminate that if we have
unique part handles?
- are we assuming an application will allocate all the part handles
before calling Zoltan for the partitioning case?  Does it actually have
to migrate entities to the processor owning a part in order to assign
those entities to that part?  Or, can a given processor only assign 
local entities to local parts?  That seems like a pretty severe 
limitation (and would only allow you to partition for increasing #'s of 
processors besides)

    +  Part handles allow identification (without communication or
O(#parts) storage) of what process the part lives on;

- so let the implementation decide whether to store the process 
alongside the part handle.  Even if an implementation decides to do 
that, that still allows retrieval of the process assignment w/o 
communication or O(#parts) storage, except in the case where you have 
O(#parts) handles, in which case you're already storing that amount of 
information.


Part III: questions/comments arising from reading current draft:

_createPartitionPar: do all processors have to assign the same handle to
the partition?  How does one designate partitions on the remote processor?

_getNumPartsOnRank, _getPartsOnRank: having unique part handles and proc
rank embedded in the part handle is insufficient for satisfying these
functions.  These functions require either O(n) storage or communication
(or assumptions about how a total # parts is distributed).  I just noted 
that that was discussed in the bootcamp and email thread.  However, 
these functions are still in the DraftInterface doc as of 5/27 (with 
comments later about this issue)



- tim


-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706