itaps-parallel removal of

[Mark Miller]

Carl Ollivier-Gooch wrote:
> 
> Mark Miller wrote:
> >
> >
> > Onkar pointed out to me that he thinks
> >
> > the decsion to remove iMeshP_getPartsOnRank from the interface
> >
> > yesterday makes it impossible for a processor to obtain the part
> >
> > handle for any part that is both a) remote and b) not a 'neighbor'.
> >
> > If you look at any of the methods that return part handles, I think
> >
> > you'll find that they all require that either the associated part is local
> >
> > or that it is the 'stored with' a 'copy' of an entity.
> >
> >
> >
> > This is bad for a parallel mesh decomposer, for example, which
> >
> > needs to be able to 'assign' entities to 'any' parts. It will need to
> >
> > be able to identify parts to 'send' entities to that are neither local
> >
> > nor neighbors of the 'initial' decomposition. I suppose one could
> >
> > argue that it could achieve this by sending entities on a 'journey'
> >
> > through all the part neighbors between the originating and destination
> >
> > parts. But, that wouldn't be too efficient.
> 
> By "parallel mesh decomposer" I assume you do -not- mean a full-up
> partitioner (which will have a list of part handles that come with the
> entities to be partitioned).

I don't think we need to get that specific about a use case scenario
to see a problem here.

If there is no function in the interface to return distantly remote
part handles of other parts (e.g. part handles that are NOT already
somehow being stored locally because they '...come with the entities
to be partitioned), then how does one obtain such part handles? I think
we can consider this question WITHOUT having to consider the specific
use case(s) that may create the need for it, unless you are trying
to argue that such use cases are too 'unusual' to be of any real
merit to consider.

> Instead, you're presumably talking about a
> scenario qualitatively similar to having a single process read a
> parallel mesh and dice it -before- the partitioner gets involved.

A streaming mesh decomposer might like to do this. But, no, I was
thinking about a use case where a mesh that has already been
partitioned and exists as parts on many processors is going to be
re-partitioned. Each processor will independently add parts for the
new partition. A sync function will ensure that each processor winds
up with part handles associated with entities that are copies of
'neighbors' and that can be obtained via the interface calls to get
those handles. However, at no time will a processor be able to
obtain a distantly remote part handle.

> 
> In this context, I could argue that the parallel read will have to do
> some outside-iMeshP communication so that the reading proc can have all
> the part handles.
> 
> Or I could argue (and this is the alternative I prefer) that it's
> possible to have part handles that:
> 
> a) are globally unique;
> b) allow identification of what process they live on;
> c) can be computed knowing only process rank and local part number; and
> d) still be be universally distinguished from pointers.
> 
> So here we go:
> 
> We know what the number of processors is (MPI communicator size or
> equivalent).  So we can easily compute the number of bits required to
> store the process rank.  Call this m.
> 
> This leaves n = 30-m bits for the number of parts per process.  That 30
> is deliberate.
> 
> Then the part handle can be an int that looks like this:
> 
> Bit
> 31 30 29 28 ... 31-m n n-1 n-2 ... 3 2 1 0
> 1  [process rank   ] [part # in process] 1
> 
> That 1 at the end is there because no pointer to a struct ever ends in
> anything but 00.  The 1 at the beginning disambiguates this from integer
> handles (which will surely be positive).  And having "only" 30 bits left
> over for rank and part # in process isn't that much of a restriction.
> Yes, I can imagine a million processes or a million parts on one
> process, but not as many as 1000 parts on each of a million processes.
> And of course 64 bit handles make questions of size moot.

Carl, there is no doubt that this or something like it will work.
The question for me is, are we placing such restrictions on a part
handle that this is the ONLY way it could in fact work? And, if so,
might it make more sense to simply admit (and sort of require that)
now? At one point I think we argued that a part handle could be
a pointer to a structure that contained above information (mpi rank
and local part #). Because, remember, it presently has to be sized
such that it fits in a void * as per its overloaded use in other
parts of the interface.

> 
> Given this approach, one can compute (from the number of parts on a
> process and the process rank) the handles of all those parts.  This way,
> you "only" have to know the number of parts on a process (which should
> be very compressible data: most will be the same...) instead of all the
> part handles.


Yes, it means the function to obtain a distantly remote part handle
could still be a non-communicating function because the handle could
be 'computed'. However, I still think that there should be an interface
call to do that rather than leaving that computation up to the client.


-- 
Mark C. Miller, Lawrence Livermore National Laboratory
email: mailto:miller86 at llnl.gov
(M/T/W) (925)-423-5901 (!!LLNL BUSINESS ONLY!!)
(Th/F)  (530)-753-8511 (!!LLNL BUSINESS ONLY!!)