itaps-parallel Diagram of set-based parallel data

[Mark Miller]

Hi Tim,

In many ways, what you describe here is very similar to a recent enhancement I
made to Silo (a scientific database file format like Exodus) to describe various sets
used in a scientific computing application. In Silo, we use the 'name' of the set(s)
as the 'clue' that a given set spans multiple processors like you use tags and tag
values.


Tim Tautges wrote:

> Hi all,
>    As promised, attached is a cartoon describing set-based parallel
> data.  Note I've stuck to the simplest example (as wisely advocated by
> Mark S).  I show the representation of the same mesh in the serial and
> 3-processor cases.  I don't show ghosted elements, for simplicity,
> though those "fit" in the model.
>
> PARALLEL
>
> In the parallel representation, each processor has a set it understands
> as "partition1" (labeled "Partn1").  In this example, on each processor,

I assume this 'understanding' is achieved via some conventional tag and
and tag value(s)? E.g. any tag named such that strncmp("Partition",tagName,9)==0?
is treated as the portion of the same set distributed over processors?

>
> that set contains one set corresponding to the part assigned to that
> processor.  The part set contains 3d entities assigned to that
> processor.  In this example, all 3d entities are objects in the
> decomposition, therefore each 3d entity is a member of exactly one Part.

So, for clarity, if we instantied this distributed iMesh on 2 processors and did
nothing to 're-decompose it'; we'd wind up with one that had a 'Partn1" set with two
'parts' sets in it and one that had a 'Partn1' set with one 'part' set in it? Are the
only members of "Partn1" sets the parts that form the partition?

> There are other entity sets defined in this mesh, including a boundary
> condition set (BC1), material sets (Matl1, Matl2), and
> application-defined sets (App1-App3).  In this example, the BC1 set
> contains faces in the mesh (2d elements), but not all of them.

Ok, it might have complicated the figure two much but it looks to me like
BC1 is in some sense 'globally defined' in that it is not decomposed for
parallel. It is not part of the "Partn1" set. Is the 'whole' BC1 the union
of the BC's on each processor? In theory, if a mesh has been decomposed
for parallel AND if that mesh contains bc's that span the 'parts' the bc's
are essentially partitioned in the same way the mesh it. I am probably
not thinking about this in the ITAPS-way.

> Matl1
> and Matl2 are meant to form a covering over 3d elements, defining the
> materials for those elements.  Application-defined sets are for
> arbitrary groupings.  Whether such "shared" sets would exist on
> processors even if there were no set contents on those processors is
> currently implementation-defined, though maybe there should be a common
> convention for that.

I assume an 'Application-defined' set can span processors using same
implict tag convention if an application so chooses?

I have the same question here I had for BC above. If we think of the
single, coherent (global) mesh (and all of its interesting subsets) as
a single coherent thing. Then, when we decompose it into pieces,
we also decompose all of its subsets into pieces (except where we
get lucky and the parallel decomposition DOES NOT cut an existing
set). So I am perplexed how we can capture the parallel decomposition
of the whole mesh *inside* "Partn1" sets without also capturing all
the interesting subsets of the mesh there? I must be thinking about this
with too much Silo baggage.

I think maybe a better way of asking my question(s) here is to ask
what it is about the "Partn1" and "part" sets that cause them to be
treated differently than you have material and bc sets? I mean why
don't material sets need to be placed in their own "MatDecomp"
set and likewise bc sets in their own 'BoundaryConds' sets?

>
>
> Some of the sets on each processor are implicitly linked with sets on
> other processors.  For example, the Partn1 set on each processor is
> linked to the corresponding sets on other procs.  For this set, that
> means that if the mesh were ever to be assembled in a single file, the
> contents of Partn1 on various processors would all go into a single set,
> which would be assigned the union of all tags on that set on all
> processors.

So, when you say this, do you mean you would actually 'knit things together'
and wind up with a single set in which all 3D entities are stored or would
you wind up with a set that contains 3 sets each of which in turn contains
the 3D entities? I figure the latter based on your picture.

> Presumably those tags would all be the same.  Similarly,
> BC1 and Matl1, Matl2 are sets that are implicitly linked and which would
> be combined.  App1-App3 are application-defined sets which would be
> considered separate, even if the mesh was written to a single file.
> Note that the decision or designation of which sets across the
> processors are really the same sets is application- or
> implementation-defined.  In MOAB, we do it based on tag and optionally
> tag value.  So, for example, any set with a "MATERIAL_SET" tag whose
> value is the same is understood to be the same set.  This is used in the
> HDF5-based file writer in MOAB, which writes a single file in parallel
> from all processors.

>
>
> SERIAL
>
> In the serial case, all sets having implicitly-defined counterparts on
> other processors have been combined.  Thus, part sets (Part1-Part3) are
> contained in the Partn1 set, and there are single BC1, Matl1 and Matl2
> sets.  App1-App3 are still separate, since they were not designated to
> be the same sets.  The Part1-Part3 sets and Matl1-Matl2 sets are
> different coverings of all 3d entities in this mesh for this example.
> The BC1 set includes a subset of 2d entities in the mesh.

So, are you saying Mat1 and Mat2 are two different material decompositions
or you saying Mat1 and Mat2 are two different materials in the same,
single, material decomposition (covering)

>
>
> DISCUSSION
>
> As I've mentioned before, I've implemented MOAB's parallel reader such
> that any collection of sets which forms a covering over the things I
> want to partition can be used to define the partition.  Thus, in this
> example, one could use Matl1/Matl2 as the part sets.  In this case,
> those sets would serve multiple purposes.  Similarly, I don't see any
> reason we shouldn't allow a given part set to be a member of multiple
> partitions.
>
> Part sets should also be allowed to include sets; in this case, maybe
> the sets are the objects in the partition.  This would be a very nice
> thing to use for e.g. hypergraph partitioning methods.
> Counter-intuitively, I would lean toward using contains relations to go
> down that hypergraph hierarchy.  If a given implementation did not
> implement Parts as legitimate entity sets, then this property could be
> used to reduce bookkeeping for adaptive applications, e.g. have the part
> contain sets which themselves contain 3d elements of a given material or
> geometric volume.  Of course, that assumes you're using sets for those
> other things...
>
> This method for representing partitions also has the following benefits:
> - allows multiple partitions to exist in the same file/instance/database
> - would allow current set operations to be used to query contents of
> parts and partitions
> - parallel view is only different when asking about parallel data; in
> other cases it looks the same as serial interface
> - other parts of RPI partition model can be embedded similarly, by using
> sets related to part sets using parent/child relations

I like these benefits very much. It seems a lot of work was put into TSTT
now iMesh interface to enable this kind of functionality. So, why not use
it?

Apologies for what I am sure are a lot of confusions regarding my interpretation
of this.

Mark

--
Mark C. Miller, Lawrence Livermore National Laboratory
email: mailto:miller86 at llnl.gov
(M/T/W) (925)-423-5901 (!!LLNL BUSINESS ONLY!!)
(Th/F)  (530)-753-8511 (!!LLNL BUSINESS ONLY!!)