[MOAB-dev] parallel itaps (more patches)

[Jed Brown]

On Mon 2008-09-15 08:55, Tim Tautges wrote:
> Thanks Jed.  I'm forwarding to itaps-parallel too.  A few notes:
>
> 1. We spoke at length among the itaps team about iMeshP last week,  
> resulting in some changes to the spec which will happen, particularly  
> with exchanging tags.  Stay tuned, but the changes will likely improve  
> things rather than make things worse.

Great, I'll watch the archives to keep up to date.

One function that is useful to me is a reduction operation on tags.
That is, MOAB's exchange_tags() and all that I've seen proposed for
iMeshP are broadcast operations (owner pushes).  Since my reduction is
still a metadata operation, it seems natural to implement at the
mesh/tag level.  (I only need MIN, but it would be no harder to support
a general reduce.)  I just started implementing this (using the
ownership data available on tags after a MOAB parallel load).

> 2. Your comments are relevant as much or more to MOAB's implementation  
> of iMeshP than they are to the interface in general, and my  

Absolutely, however the type-correctness in the headers is certainly an
interface issue, regardless of whether the official header adopts the
incomplete types rather than void*.  Should I send the iMesh.h part of
my patch to the tstt-interface list?  (I notice you didn't forward the
patch.)

> implementation is partial at best.  I don't do anything special with  
> parts or partitions, so that support is questionable.  I will have full  
> support by the SC08 conference in November, but probably won't start in  
> earnest until Oct 1.

Cool, the MOAB exchange_tags() only partially works for me.  I'm still
trying to identify exactly which entities are not getting updated.  Most
are, but for a particular mesh partitioned 4-ways, the entities on rank
1 owned by rank 0 and the entities on rank 2 shared with rank 1, 3 and
owned by rank 0 are not updated.  In the latter case, the copies on 1
and 3 were updated as well as the entities shared exclusively with 0.  I
browsed the implementation, but not carefully.

Do you have a recommendation for storing irregular data associated with
entities?  I have variable spectral order throughout my mesh and would
like to be able to save the full state.  Since a high-order hex has
hundreds of dofs associated with it, it's not ideal to just make a tag
with order equal to the maximum order.  I can make a collection of tags
with maximum orders 4,16,64,256,1024 (for instance) and just choose
element-by-element the smallest that works.  An alternative is to store
the vector separately and just keep a size and offset for each entity.
This is nicer in some ways because the state file is very simple and can
easily take advantage of parallel IO, but it does require a two-stage
load process.

> 3.  We're still talking about load options, but in the meantime you're  
> fine using the options below.  They won't be interoperable, but I'm  
> hoping the interoperable version will look something like it (started  
> discussing this on Fri).

What are you using to write a parallel mesh?  Apparently this is not
implemented per-se, so I've just been writing a separate file for each
process.  With an ownership tag, this could easily be recombined in
serial, but I haven't written that utility.

Thanks,

Jed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <https://lists.mcs.anl.gov/mailman/private/moab-dev/attachments/20080915/040c956c/attachment.pgp>