itaps-parallel One particularly difficult thing in one of the use cases...

[Tim Tautges]

Sorry, I was trying to err on the side of conciseness (in contrast to my writeup of the use cases as solved by MOAB, 
which is anything but...).

On 06/02/2010 08:24 AM, Mark Beall wrote:
> I'm either not clear on the example or unsure of how you would
> accomplish step 2 with iMeshP anyhow.
>
> For step 2 are you saying that you're creating ghosts of entities from
> the 1st column on each other processor in the row? (I don't know if by
> "share" you mean something else).
>

You're sharing all the entities in a given subdomain across all processes in that row.  In my writeup, I specify using 
iMeshP_exchEntArrToPartsAll, assuming that you'd have to repeat entities in the entity list, once for each target 
Process/PartID.

The full description of that step, from my writeup (yet to be sent in full), is:

-------------------------------
C. Create Partitions Psi and Parts Psioj, based on one communicator
    for each row in the table comm_row_i; put the mesh from Po1si into
    Psio1; and copy the mesh from Mi1 to Mij across Psi

The application has established one communicator for each row of the
table.  In this step, the mesh in the first column of each row (on
Process Po1si) is put into the new Part in Psi, then copied across the
row, or across the Partition Psi, but retaining the first column Parts
Psio1 as owners of the copied mesh.

1. Create a Partition Psi on Processes Mij, using
    iMeshP_createPartitionAll(comm_row_i).

2. Create Parts Psio1, using iMeshP_createPart(Psi)

3. Add all entities in si to Part Psio1, using iMesh_addEntArrToSet(Psio1)

4. Copy all entities to all Processes Mij across Psi,
    iMeshP_exchEntArrToPartsAll(Psi)


- tim

> I don't see that iMeshP_createGhostEntsAll could even be used in that
> way, it creates ghosts of a certain number of layers from the partition
> boundary. It also requires a partition into which to create the ghost
> entities.
>
> iMeshP_addGhostOf could be used, it requires the target part.
>
> mark
>
> On Jun 1, 2010, at 9:52 PM, Tim Tautges wrote:
>
>> Hi all,
>> Here, I attempt to describe one of the particularly difficult parts of
>> one of the use cases. I think this at least exposes a potential
>> problem in the currently specified iMeshP; I think it also
>> demonstrates the problem with entities always having to be in parts,
>> but that's more of a subjective statement.
>>
>> The use case is radiation transport. In a nutshell, consider processes
>> arranged in a 2d array, with each column representing an angle, and
>> each row a subdomain in the spatial partition. The spatial partition
>> of the mesh will be distributed across all processes in a column; a
>> given spatial subdomain is copied onto each process in a row.
>>
>> Initializing the mesh onto this processor arrangement is done in 3 steps:
>> 1. load the mesh onto the 1st column, in a domain-decomposed fashion
>> typical of other DD-based codes
>> 2. share the mesh from the 1st column across a row of processes
>> 3. establish sharing relations in the other columns
>>
>> It's in step 3 that I see the problem. Here, the mesh for a spatial
>> subdomain is already represented in the iMesh Instance on each
>> process, but the Partition representing the column hasn't been created
>> yet. After you create that Partition, and a Part in that Partition,
>> you need to assign the mesh from the Part in the row-based Partition
>> into that Part in the column-based Partition. How do you do that? The
>> function iMeshP_exchEntArrToPartsAll is a collective call, and implies
>> that you're moving entities from one Part to another. But, the
>> entities we're talking about here aren't part of any Part in that
>> column Partition. I'd prefer to use iMesh_addEntToSet, but I'm
>> guessing that would break other implementations.
>>
>> This example also demonstrates the need either for another function,
>> to negotiate shared entities between Parts, or to expand the
>> definition of iMeshP_createGhostEntsAll to include the functionality
>> (the latter would be most natural for MOAB, since the same function is
>> used in MOAB's parallel stuff to do either; I distinguish by allowing
>> the # layers specified to be zero, in which case you're requesting the
>> resolution of shared entities at an interface).
>>
>> - tim
>>
>> --
>> ================================================================
>> "You will keep in perfect peace him whose mind is
>> steadfast, because he trusts in you." Isaiah 26:3
>>
>> Tim Tautges Argonne National Laboratory
>> (tautges at mcs.anl.gov) (telecommuting from UW-Madison)
>> phone: (608) 263-8485 1500 Engineering Dr.
>> fax: (608) 263-4499 Madison, WI 53706
>>
>
>

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706