[MOAB-dev] Re: parallel read bug, interface awkwardness

[Tim Tautges]

Hi Jed,
   I agree that it's a real bug, and that we shouldn't assume things 
about the partition which are unnecessary to assume.  I'm in a middle of 
a milestone exercise, which ends next Wed, and will get to it right 
after.  If you can't wait, I think the issue is in 
MBParallelComm::resolve_shared_ents and where it calls tag_shared_ents 
and create_interface_sets.  tag_shared_ents builds or adds to the map 
proc_nranges, where they key for the map is the list of procs sharing a 
given set of entities, and the value is the range of entities. 
create_interface_sets takes these key/value pairs, builds an entity set 
for each, and assigns to the lowest-rank proc in the key ownership of 
that set.

To fix, you'll need to assign faces to the owning proc as assigned by 
the partitioner.  One complication is that may produce keys which aren't 
already in the map.  Another way to do this would be to get all the 
interface faces and their assigned processors and build the key/value 
pairs for those, before tag_shared_ents is called.  The block of code 
which sets the NOT_OWNED part of the pstatus tag is at 
MBParallelComm.cpp:2616 (in svn version 2078).  That will need to be 
changed to use the partitioner-determined owner rather than just taking 
the min rank.

If you don't have time to fix this on your own, no problem (you've done 
quite a bit already, which I appreciate).

- tim


Jed Brown wrote:
> On Mon 2008-09-22 10:25, Jed Brown wrote:
>> I have identified a MOAB parallel read bug.  It appears that if all
>> nodes of an edge or face are shared, MOAB sometimes assumes that the
>> edge/face is also shared.  This is incorrect since the face might
>> actually be a boundary face.  I think that if
>>
>> 0 --- 1 --- 2
>> |  A  |  B  |
>> 3 --- 4 --- 5
>> |  C  |  D  |
>> 6 --- 7 --- 8
>> |  E  |  F  |
>> 9 --- 10 -- 11
>>
>> appears on a boundary where proc 0 owns all faces except face D which is
>> owned by proc 1.  Then proc 0 will own all edges except 5-8, but proc 1
>> will think that 5-8 and face D are also owned by proc 0.  The attached
>> tarball demonstrates the issue.  If you unpack and run `make break' you
>> will see that the shared sets don't agree.  Run `make repartition' to
>> produce a new partition (likely different because mbzoltan is
>> nondeterministic).  I have run `make repartition break' a few times and
>> there are always sets which don't agree.
> 
> To follow up on this, a recent patch has greatly improved the quality of
> mbzoltan partitions.  At least for simple cases, these partitions do not
> have the property mentioned before.  However, the read bug still exists
> for these `pathological' partitions such as the one attached in my
> previous message.  To reproduce, just unpack and run `make break'.
> Hopefully someone more familiar with the parallel read knows how to fix
> this.
> 
> A sufficiently complex mesh may not be partitionable in a way which
> avoids this so it's really not okay to assume that the process which
> owns the vertices also owns the entity; it may not be adjacent to any
> owned region.  This matters to me because edges and faces are used
> explicitly as part of my function space.
> 
> Jed

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706