itaps-parallel Mesh Instances

[Tim Tautges]

Devine, Karen D. wrote:
> All:  Thanks for your emails.  I admit I am getting more confused than
> before by this discussion.  So let me try to get a clear answer on the
> simplest serial cases before we move to parallelism and iPartition.
> 
> Suppose a serial multiphysics application has two Exodus files with one mesh
> description in each file.  The application wants a single ITAPS
> implementation (e.g., MOAB) to manage both meshes.  It wants to do
> completely different operations on each mesh.  Suppose one mesh is a 3D
> hexahedral mesh and the other is a 2D quadrilateral mesh.
> 
> A) Should the application create two mesh instances, and ask the
> implementation to load meshInstance1 from ExodusFile1 and meshInstance2 from
> ExodusFile2?  Or ...
> B) Should both Exodus files be loaded into the same instance?
> 

I believe most use cases would call for loading both meshes into the 
same instance, distinguishing them using entity sets (for example, MOAB 
already creates and populates a set with the mesh read from a given 
file).  This would simplify using tools e.g. to find spatial proximity 
of elements in either mesh, to intersect the skin of one with the other, 
etc.

> For case B, what answer does the following iMesh function return?  How does
> this function determine which mesh it is dealing with?
> 
>    void iMesh_getGeometricDimension(iMesh_Instance instance,
>    int *geom_dim, int *err);
> 
> I assumed that since we pass the mesh instance to each iMesh function,
> implementations can use it to determine which mesh they are to query.  So
> this function would return appropriate info for case A.
> 

This is a function which applies to the entire instance, and in many 
ways is application (and file) dependent.  Carl has separate 
implementations for 2 and 3 dimensions; in MOAB it depends on what kind 
of mesh is loaded, maybe (or is always 3d, likely).  If the file format 
supports it, one could embed the geometric dimension as a tag either on 
the file set or a subset containing just the Exodus SHELL-type elements 
(or is it QUADs which are 2d, I forget?)

> Yes, this example is contrived, but it illustrates my point.  I could think
> of more complex examples, but they would be harder to explain.  This example
> doesn't have to deal with parallelism (yet), or with the multiplexer (since
> a single implementation is used).
> 

S'ok, welcome to the wacky world of iMesh.

One thing that might be helpful, coming from a (former) Sandian, is to 
try to think of meshes in broader terms than what can be stuffed into an 
Exodus file (Mark described how Silo and SAF do things, for example). 
Exodus uses blocks both for describing the elements themselves as well 
as their material type, which can be limiting sometimes (and that's 
ignoring sidesets, which are even worse).

- tim

> Thanks for teaching me the mysterious ways of the iMesh.
> 
> Karen
> 
> 
> On 11/19/07 1:15 PM, "Tim Tautges" <tautges at mcs.anl.gov> wrote:
> 
>> Hi all,
>>
>> First, and I think this is most important: in reality, we're targeting
>> the API at two primary uses (and several others): a) applications
>> accessing mesh data stored in some implementation of the API, and b)
>> applications wrapping their representation of mesh for iMesh-enabled
>> tools to work on.  Both those uses lean towards single-iMesh-instance
>> applications, and up to now we've focused heavily on that (and
>> intentionally, because the question has come up before).  I think this
>> is a really important issue to resolve again at a high level; that is,
>> are we developing these higher-level interfaces (like parallel mesh, but
>> also fields eventually) to explicitly treat multiple underlying
>> implementations of the same iXxx interfaces.  I'd still assert that from
>> an interface point of view, we can do most of what applications will
>> want while maintaining our primary focus on the single-instance case.  I
>> also assert that we're not explicitly ruling out the multi-instance
>> case, which can still be done (though not quite as easily).
>>
>> The single instance vs. multiple instance question pertains to both
>> serial and parallel applications.  Treating parallel mesh adds another
>> hierarchy to the data at least, even for a single iMesh instance per
>> processor.  Whether to take the multiple iMesh instances per processor
>> into iPartition explicitly, or to accomplish that by assuming some
>> multiplexer that iPartition doesn't know the details of, is the question.
>>
>> I think there are two primary reasons for keeping iPartition focused on
>> one iMesh instance per processor.  First, from a complexity point of
>> view, keeping the interfaces separated makes the sum less complex.
>> Second, most of our current applications are single-instance and likely
>> to stay that way, and for the future ones (esp. multiphysics), it's
>> still not clear that treating multiple iMesh instances per iPartition,
>> without a multiplexer, would be the best way to structure a solution.
>>
>> For example, I'm helping to develop a parallel multiphysics reactor
>> simulation code at ANL, composed from existing components for heat
>> transfer & fluid flow, neutronics, and soon structural mechanics.  Each
>> of these components already operates in parallel, on its own mesh.
>> Among other things, one of the physics-based things we have to do is map
>> solutions between meshes, possibly with those meshes having completely
>> different distributions on possibly non-overlapping sets of processors.
>>   To map solutions between meshes efficiently and accurately, you start
>> needing lots of mesh-based support, things like data sorting and
>> searching and interpolation for example.  That motivates the need for an
>> iMesh instance in which the solution remapper can construct its own data
>> structures.  If one wants this remapping component to be interoperable,
>> it should be constructed on top of iPartition also.
>>
>> Now, by definition as iPartition is currently being discussed (in the
>> multiple instance case), the application still has to deal with multiple
>> instances of iMesh, whether iPartition does or not.  Having multiple
>> instances treated by iPartition isn't going to relieve the application,
>> or the solution remapper, from having to also deal with that issue.
>> Given that, the question becomes can we accomplish what we need to with
>> iPartition using a single iMesh instance, with the various applications
>> we have in mind, and is that solution easier without being too
>> restrictive.  So far, I think the answer is yes.
>>
>> There's a lot more detail I could go into and won't, given the length of
>> this message already.  I'll also add a few specific replies to the
>> various email discussions as well.
>>
>> - tim
>>
>>
>>
>> Parallel mesh: this adds
>>
>> Devine, Karen D. wrote:
>>> In yesterday's meeting, Lori asked me to add our understanding of iMesh
>>> instances to the background and terminology section of the combined
>>> document.  I am unable to do so until we come to some agreement on what mesh
>>> instances are and how they are to be used.
>>>
>>> Let's start with getting agreement for serial applications.
>>>
>>> In the iMesh user guide, I cannot find a definition of what a mesh instance
>>> represents.  To me as a new(ish) ITAPS user, it looks like it represents a
>>> single mesh.  If our interface were a C++ interface, I would say that iMesh
>>> is a class, and that we'd create one iMesh object for each mesh we used in
>>> the application.  Then we'd invoke methods on each iMesh object as needed.
>>> For examples, in a loosely coupled physics implementation on two meshes,
>>> we'd invoke meshOne's methods to do the first set of physics, and meshTwo's
>>> methods to do the second set of physics.
>>>
>>> In our C interface, we can't invoke methods on an object.  What we do
>>> instead is what most people do when writing an object-oriented C interface:
>>> they pass the object to the interface functions.  It is like passing C++'s
>>> "this" pointer to C functions.
>>>
>>> I think this is a natural way for users of multiple meshes to view the iMesh
>>> instance, and I don't see anything in the documentation to contradict it.  I
>>> also think we need to allow multiple iMesh instances to allow multiple
>>> implementations to be used at the same time via the multiplexer.
>>>
>>> Let's get this serial scenario understood before we move on to parallel.
>>>
>>> Karen
>>>
>>> ------ Forwarded Message
>>> From: Mark Miller <miller86 at llnl.gov>
>>> Organization: Lawrence Livermore National Lab
>>> Date: Fri, 16 Nov 2007 00:21:51 -0800
>>> To: "itaps-parallel at mcs.anl.gov" <itaps-parallel at mcs.anl.gov>
>>> Subject: itaps-parallel comments on today's meeting
>>>
>>> I think it is important to understand in detail Tim's insistence that
>>> there should be only one iMesh_instance object per 'parallel task'
>>> even to the point of aglomerating multiple different meshes of
>>> an application into that iMesh_instance and then distinguishing
>>> between them via a multiplexer. I think I understand some of his
>>> rationale for this but not all of it. And, I am sure I don't
>>> understand how multiplexing helps as my understanding is that
>>> the multiplexer is the iMesh instance (at least in Jason's example
>>> he sent us).
>>>
>>> ------ End of Forwarded Message
>>>
>>>
>>>
> 
> 
> 

-- 
================================================================
"You will keep in perfect peace him whose mind is
  steadfast, because he trusts in you."               Isaiah 26:3

             Tim Tautges            Argonne National Laboratory
         (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
         phone: (608) 263-8485      1500 Engineering Dr.
           fax: (608) 263-4499      Madison, WI 53706