itaps-parallel One particularly difficult thing in one of the use cases...

[Mark Beall]

I'm either not clear on the example or unsure of how you would  
accomplish step 2 with iMeshP anyhow.

For step 2 are you saying that you're creating ghosts of entities from  
the 1st column on each other processor in the row? (I don't know if by  
"share" you mean something else).

I don't see that iMeshP_createGhostEntsAll could even be used in that  
way, it creates ghosts of a certain number of layers from the  
partition boundary. It also requires a partition into which to create  
the ghost entities.

iMeshP_addGhostOf could be used, it requires the target part.

mark

On Jun 1, 2010, at 9:52 PM, Tim Tautges wrote:

> Hi all,
>  Here, I attempt to describe one of the particularly difficult parts  
> of one of the use cases.  I think this at least exposes a potential  
> problem in the currently specified iMeshP; I think it also  
> demonstrates the problem with entities always having to be in parts,  
> but that's more of a subjective statement.
>
> The use case is radiation transport.  In a nutshell, consider  
> processes arranged in a 2d array, with each column representing an  
> angle, and each row a subdomain in the spatial partition.  The  
> spatial partition of the mesh will be distributed across all  
> processes in a column; a given spatial subdomain is copied onto each  
> process in a row.
>
> Initializing the mesh onto this processor arrangement is done in 3  
> steps:
> 1. load the mesh onto the 1st column, in a domain-decomposed fashion  
> typical of other DD-based codes
> 2. share the mesh from the 1st column across a row of processes
> 3. establish sharing relations in the other columns
>
> It's in step 3 that I see the problem.  Here, the mesh for a spatial  
> subdomain is already represented in the iMesh Instance on each  
> process, but the Partition representing the column hasn't been  
> created yet.  After you create that Partition, and a Part in that  
> Partition, you need to assign the mesh from the Part in the row- 
> based Partition into that Part in the column-based Partition.  How  
> do you do that?  The function iMeshP_exchEntArrToPartsAll is a  
> collective call, and implies that you're moving entities from one  
> Part to another.  But, the entities we're talking about here aren't  
> part of any Part in that column Partition.  I'd prefer to use  
> iMesh_addEntToSet, but I'm guessing that would break other  
> implementations.
>
> This example also demonstrates the need either for another function,  
> to negotiate shared entities between Parts, or to expand the  
> definition of iMeshP_createGhostEntsAll to include the functionality  
> (the latter would be most natural for MOAB, since the same function  
> is used in MOAB's parallel stuff to do either; I distinguish by  
> allowing the # layers specified to be zero, in which case you're  
> requesting the resolution of shared entities at an interface).
>
> - tim
>
> -- 
> ================================================================
> "You will keep in perfect peace him whose mind is
>  steadfast, because he trusts in you."               Isaiah 26:3
>
>             Tim Tautges            Argonne National Laboratory
>         (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>         phone: (608) 263-8485      1500 Engineering Dr.
>           fax: (608) 263-4499      Madison, WI 53706
>