itaps-parallel Notes from today's meeting
Mark Shephard
shephard at scorec.rpi.edu
Mon Apr 19 05:15:02 CDT 2010
Two quick comments/questions:
Is there a conference call today, Monday? If yes, what time?
The issues of doing a contact problem in parallel are really much the
say as doing mesh adaptivity in parallel with a form of nonmanifold
model. The only new item past that of a 2-manifold situation is keeping
track of the contact face interactions. There is absolutely no problem
doing this within in the concept of a single partition of the mesh. On
the other hand, considering how to track an evolving contact interface
through the use of multiple partitions of the mesh would actually add
substantial complexity to the mesh up-date and dynamic load balancing
processes.
Mark
Carl Ollivier-Gooch wrote:
> Tim Tautges wrote:
>
>> I think Jason's argument is important here: the easy things should be
>> easy, and more complicated things possible. I think this is related
>> to Seegyoung's point about having a single API. I think what this
>> boils down to is an assertion that we should think primarily in terms
>> of one iMesh Instance per process. For those applications wanting to
>> use multiple instances, that implies communication between the
>> instances at least conceptually, if not in the implementation also.
>> What this boils down to even further is that Part and iMesh instance
>> are synonymous. The issue of an entity assigned to exactly one Part
>> morphs to one of representation vs. ownership: an entity can be
>> represented in more than one instance (e.g. faces on part boundaries,
>> in an element-based partition), but can be owned by only one of those
>> instances.
>
> Personally, I'm very much in favor of the part = instance
> interpretation, for reasons already outlined.
>
> In addition, I can easily see how we'll be able to implement this,
> whereas with multiple parts per instance, the easiest approach would be
> for us, effectively, to have multiple of our current instances inside a
> wrapper; yuck!
>
> At a higher level, this also makes the whole parts-as-special-sets
> argument moot, saving us from possibly re-visiting that. If this is the
> way we decide to go, we'll want to do an audit of iMeshP functions,
> because part handles will now be synonymous with iMesh instance handles.
> I suspect that a whole lot of functions either go away or get simpler.
> This is not a bad thing.
>
>> One of the reasons we initially thought in terms of multiple parts per
>> process was to handle over-partitionings, where a process could be
>> assigned multiple parts, e.g. for better load balancing, and
>> flexibility in what's stored with a mesh or how it's stored. I think
>> that concept is an artifact of how a partitioning is stored with the
>> mesh (or how the mesh is stored in files to reflect a partitioning).
>> From an application's point of view, in most cases it's still going to
>> want to access the mesh from a single instance on a given process, no
>> matter how that mesh was loaded initially. That's also how the mesh
>> looks from most applications' points of view. The use case of
>> multiple parts, with shared interface entities duplicated, is mostly
>> handled by having separate entities in the database, with special
>> relations between them implying that they're the same.
>
> I can see ways around the over-partitioning problem, though. Most
> obviously, simply have multiple threads per processor, so that the
> number of processes still matches the number of parts. Or an app or
> implementation (or even the interface) could specify how to merge parts
> to get the right number at run time, as Tim hints at.
>
> Eventually, in the hyperparallel world that's coming, apps and
> implementations may both have to be written to be multithreaded
> shared-memory processes within a node (OpenMP?) with communication
> between nodes (MPI?). This isn't in conflict with the notion of exactly
> one part (and iMesh instance) per process; it just requires more careful
> programming of apps/services that use the interface.
>
>> One of the complications that arises from having one instance per
>> process (and one part per instance) is how do you repartition, or
>> partition in serial. I think that concept is handled by the notion of
>> initializing a given parallel mesh either a) from a collection of
>> collections, or a partition of parts, stored with and loaded with the
>> mesh, or b) communication from one instance to another, using parallel
>> migration of entities. Of course, a) is easily handled using sets, if
>> you have them; if you don't, it's just as easily handled using
>> multiple files. It also naturally handles establishing a parallel
>> mesh based on one of several "partitionings" (collection of
>> collections) stored with a mesh, e.g. using the geometric volumes,
>> material types, or true partitions (generated by a partitioner).
>
> This is a problem we always had, though, didn't we? I mean, with
> multiple instances per process, repartitioning to use a new instance
> would require copying the whole mesh, including adjacency, sets, tags,
> etc. Do-able, but non-trivial, especially (say) tags with entity handle
> value.
>
> I don't have any conceptual issues in principle with the notion of
> multiple partitionings of the same entities. How an implementation (or
> app) chooses to store the inactive partition is obviously going to be
> implementation-dependent (sets vs tags vs classification vs ???). This
> potentially makes it tricky to define an API for switching between them.
>
> Also, what about the different, more subtle case where we -need- two
> partitions active at once? One use case is a contact problem, where the
> 3D mesh and 2D surface are partitioned separately (I'm assuming this is
> done, to get better load balance?). If you're also doing mesh
> adaptation as you go (likely), now those 2D faces on the interface have
> owners in the 3D partition (okay, some implementation don't represent
> them explicitly, but it's safe to say that a face interior to a part is
> going to be owned by that part...). I see two alternatives.
>
> 1. The easiest way to handle this in the current interfaces, as far as
> parallel stuff goes, is to have two separate meshes (a 3D mesh and a 2D
> manifold mesh), with tags or iRel (or something similar) keeping the
> respective entities associated with each other. This implies two iMesh
> instances per process, one each for the 3D and 2D manifold meshes, which
> isn't in conflict with other stuff in this thread.
>
> 2. You could also have two partitions active within a single instance.
> I'm not quite sure how this would look, even conceptually. You'd have
> the full 3D mesh for the (3D) part, plus a bunch of extra faces (and
> verts, at least) for the bits of the 2D manifold mesh that isn't already
> resident. The problem with this is that you've got to strictly
> segregate things so that iteration over (say) faces gets you what you
> wanted. Yes, I know, sets will do this, but it seems cumbersome...
>
> I can't say I'm crazy about either of these alternatives, nor am I
> invested in either one. Hopefully, someone can come up with something
> better. For what's it's worth, I think this problem is reasonably
> (completely?) independent of whether in "normal" single mesh contexts we
> require exactly one part and iMesh instance per process.
>
> Having said that, I'll almost certainly be missing tomorrow's telecon. I
> will, of course, be happy to kibitz afterwards by email. :-)
>
> Carl
>
More information about the itaps-parallel
mailing list