itaps-parallel Notes from today's meeting

[Mark Beall]

On Apr 12, 2010, at 2:46 PM, Devine, Karen D wrote:
> The first big question:  Can an entity exist in an iMesh_instance  
> that has a
> partition without being owned by any part?

First, a disclaimer: somewhere in this I'm going to write partition  
instead of part (since what you call "parts" we call "partitions",  
what you call a "partition" we call a "partitioning"). I will try to  
avoid that.

It seems to me that a, not unreasonable, implementation of a  
partitioned mesh is one where each part is equivalent to a mesh (when  
I say "mesh" in here I mean "mesh instance" in the iMesh sense). This  
is the way that our partitioned mesh works and, I believe, pretty much  
how RPI's works. So we have 2 out of (how many?) iMeshP  
implementations that are designed this way. From here on I'm going to  
assume that this is a reasonable implementation, so my conclusions are  
only valid if you agree with that.

The pros of this implementation:
- Since there is already a way of knowing what entities are in a mesh,  
there is zero overhead, for the single partition case, of knowing what  
entities are in a part. I think this is an important consideration  
since the vast majority of applications won't use multiple partitions.
- There is no need for functions to loop over the contents of a part,  
as there are already those for a mesh - I would think that Tim would  
like this approach for this reason - smaller API :)
- Code that works in serial on a mesh can be used for anything that  
can be done on just the interior of a part (or on the entire part if  
it doesn't modify the part boundary)
- Since operations to create mesh entities are done on a mesh, there  
isn't ever any question on what part an entity belongs to.

The cons of this implementation:
- It's not obvious how to deal with multiple partitions (answer to  
this in second question below)
- entities on the part boundary must be duplicated even if they are on  
the same process
- (please feel free to add to this list since I can't think of any  
others)

Some conceptual differences of this implementation:
- rather than taking entities out of a part and putting them in  
another, you change their part ownership by telling them (or something  
doing the repartitioning), "I'm going to want you to be in this other  
part later". Then after you've done that for all the entities (in a  
consistent way), you repartition the mesh. The nice thing about this  
is that there is never a point in time (from the user perspective)  
that the mesh and the parts are inconsistent in any way.

For this kind of implementation, allowing a situation where a mesh  
entity is not in a part is equivalent to it not being in a mesh. While  
one could certainly have implementations where that is allowed, in our  
implementation - which I think is consistent with the intents of iMesh  
- the mesh owns the mesh entities (in the OO sense - if the mesh is  
deleted it takes all of the mesh entities with it).

As was mentioned in the paragraph I deleted below, when we have  
multiple parts per process, we have multiple meshes per process one  
per part. We can certainly make those look like a single  
iMesh_instance through the interface. In reality we could also have an  
additional hidden mesh that was the home for any mesh entities that  
aren't on a partition. The downsides to that are a) it's relatively  
expensive to move entities from one mesh to another for us (you have  
to copy it and delete the old one - which really gets messy when  
something off-process might be referring to it), b) there are rules  
that don't allow an entity to attach to lower order entities in a  
different mesh (for very good reasons).

This isn't an answer to "the first big question" but is a explanation  
of why it's a problem for some implementations.

>
> The second big question:  How should we enable multiple partitions  
> of a
> single mesh in iMeshP?

My understanding from what's been discussed on this mailing list is  
that it's expected that different partitions don't have any  
correlation to each other (in other words, it's the rare exception  
that two different partitions will result in entities being on the  
same process in both). If you don't agree with that, then, again, my  
conclusions based on that are invalid.

If partitions are not correlated then any entity in more than one  
partition must effectively be copied for each additional partition  
they are in. The only way for an entity to exist on more than one  
process is for there to be more than one copy of it (if someone has  
cases where there are multiple partitions but they don't actually need  
to know any information about the entities in one - other than that  
they are in it - that would be a counter example to this).

Given this then a second (or nth) partition is really just a partition  
of a copy of a mesh (or a subset of entities in the mesh). So why try  
really hard to hide that fact?

I've heard Mark S. say that there is a need for another mechanism for  
dividing up a mesh (I'm purposely avoiding the word partitioning here)  
on these currently mythical computers that will exist in 10 years.  
It's going to be scary if there are meshes, partitions and parts, and,  
umm, divisions and divs ...

> The third question:  Should iMeshP_destroyPart should return an  
> error if the
> part to be destroyed is not empty?
> The answer to this question depends, to some extent, on our answer  
> to the
> first big question above.
> Cons:  Requires users to explicitly remove entities from parts before
>       deleting the parts.  This operation requires three function
>       calls:  iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with
>       part handle), iMeshP_destroyPart.
> Pros:  Prevents users from shooting themselves in the foot.
>       Users would possibly call iMeshP_getEntities and
>       iMesh_rmvEntFromSet anyway to do migration to new part.

This becomes a moot point in my world...

mark