itaps-parallel One particularly difficult thing in one of the use cases...

[Tim Tautges]

Hi all,
   Here, I attempt to describe one of the particularly difficult parts of one of the use cases.  I think this at least 
exposes a potential problem in the currently specified iMeshP; I think it also demonstrates the problem with entities 
always having to be in parts, but that's more of a subjective statement.

The use case is radiation transport.  In a nutshell, consider processes arranged in a 2d array, with each column 
representing an angle, and each row a subdomain in the spatial partition.  The spatial partition of the mesh will be 
distributed across all processes in a column; a given spatial subdomain is copied onto each process in a row.

Initializing the mesh onto this processor arrangement is done in 3 steps:
1. load the mesh onto the 1st column, in a domain-decomposed fashion typical of other DD-based codes
2. share the mesh from the 1st column across a row of processes
3. establish sharing relations in the other columns

It's in step 3 that I see the problem.  Here, the mesh for a spatial subdomain is already represented in the iMesh 
Instance on each process, but the Partition representing the column hasn't been created yet.  After you create that 
Partition, and a Part in that Partition, you need to assign the mesh from the Part in the row-based Partition into that 
Part in the column-based Partition.  How do you do that?  The function iMeshP_exchEntArrToPartsAll is a collective call, 
and implies that you're moving entities from one Part to another.  But, the entities we're talking about here aren't 
part of any Part in that column Partition.  I'd prefer to use iMesh_addEntToSet, but I'm guessing that would break other 
implementations.

This example also demonstrates the need either for another function, to negotiate shared entities between Parts, or to 
expand the definition of iMeshP_createGhostEntsAll to include the functionality (the latter would be most natural for 
MOAB, since the same function is used in MOAB's parallel stuff to do either; I distinguish by allowing the # layers 
specified to be zero, in which case you're requesting the resolution of shared entities at an interface).

- tim

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706