itaps-parallel reasons against part=mesh instance

[seol at scorec.rpi.edu]

Both Tim and mark's argument make sense to me. Furthermore, I suggest to
get rid of any of our assumption/restriction on the parallel computation,
such as machine type (shared memory vs. distributed) and algorithmic
design (SIMD vs. MIMD) in our further discussion on iMeshP design (the
design document of iMeshP is really needed just as iMesh and iGeom so
making the .h file is a relatively trivial work and not time-consuming
based on our agreement on the design).

For the support of massively parallel computation, instead of taking one
of two different views of "part" below - runtime notion vs. unit of
partitioning, it might be desirable to support both views in the ITAPS.
iow, allowing the multiple parts to be processed all at the same
time or one part at a time where the part is the unit of mesh
partitioning. And I believe that's what it mean by true parallel mesh
management.

In any case, we will need to come up with a method(or library) running
underneath so it frees us from the low-level worries on the thread
management no matter what the distributed computation is (synchronous,
asynchronous, partially synchronous, shared memory, distributed, etc.).

Seegyoung

>
> On Apr 22, 2010, at 10:39 AM, Tim Tautges wrote:
>
>>
>>
>> Mark Beall wrote:
>>> I don't see how having part=mesh instance and having more than one
>>> part per process implies multiple threads. Given the current
>>> interface, that would seem to be true since the mesh instance is
>>> passed in as the "context" of all the collective calls since it
>>> basically means "all of the mesh on this process", it would just be
>>> a matter of having something else that means the same thing.
>>
>> Well, that implies then that iMeshP interacts with multiple iMesh
>> instances, right?  That's substantially different from the way both
>> FMDB and MOAB have been designed up to this point.
>
> My understanding was that FMDB is similar to our implementation, but
> that understanding could be wrong.
>
>> It also implies that iMesh instances are thread safe, which isn't a
>> guaranteed thing by any means (though MOAB is).
>
> Actually, exactly the opposite. It allows you to use multiple threads
> without the iMesh instances being thread safe.
>
>> The deeper issue here, though, is that you don't want to have
>> multiple copies of entities local to a process, both for memory and
>> for logistical reasons.
>>
>> Memory-wise, the amount of memory per core is decreasing or holding
>> steady; BGP and Q have / will have around 512 MB/core.  Memory
>> accounts for over half the cost of these large machines, I think, so
>> increasing that isn't a low-cost solution.
>
> What would be a typical number of elements per core? If I'm at 100,000
> elements per core, then I'm using a few 10s of MB for the mesh and
> copying the boundary is going to be a tiny fraction of that. If the
> partition sizes are smaller the fraction for copying the boundary goes
> up, but the total memory is a tiny fraction of the amount of memory
> per core.
>
>> Logistically, we've designed iMeshP such that each partition has a
>> 1-1 association with an MPI communicator.  There are multiple
>> situations where one wants to communicate with different groups of
>> processes, from the same instance.  I've pointed out the radiation
>> transport application already; I can name a few others if you're
>> interested.  Sure, we can make wholesale changes to iMeshP based on
>> different assumptions, but then there are a few other decisions I'd
>> like to revisit.
>
> I guess I don't see any reason that would have to change.
>
>> Along the same lines, since an
>>> iMeshP_PartitionHandle is associated with a single iMesh_Instance
>>> why is it even necessary to pass both into those functions, the
>>> mesh instance is redundant since there is only one that can be the
>>> correct one, right?
>>
>> We have not assumed that a partition knows about the instance.
>> That's kind of like STL container iterators not knowing about the
>> container itself.  If we had language-specific wrappers, the iMesh
>> instance would likely be the class in C++; it's that way in the
>> python interface we've developed at UW.
>
> What harm would there be in having the partition know about the
> instance? It would make a cleaner interface and avoid a source of
> errors from people passing in the wrong instance. Seems like a better
> choice to me.
>
>> - tim
>>
>>> mark
>>> On Apr 21, 2010, at 12:03 PM, Tim Tautges wrote:
>>>> Some of the following argument depends on how partitions are
>>>> handled.  However, assuming the partition is also associated with
>>>> a mesh instance (with the partition in each mesh instance
>>>> coordinated with those on other instances over the parallel
>>>> job)... using one part per instance, and multiple instances on a
>>>> given process, implies multiple threads of control, since many of
>>>> the iMeshP functions are collective calls.  Single threads of
>>>> control are far and away the most common mode for running codes at
>>>> scale right now, and I assert will continue to be for some time
>>>> (5-10 yrs).
>>>>
>>>> Also, for the mode where an application is providing an iMeshP
>>>> implementation on top of its data structure so it can use services
>>>> implemented on iMeshP, I think the restriction of one part per
>>>> instance means that these apps will always restrict themselves to
>>>> one part per process.  I think your application is of this type.
>>>> So, I'd much rather have this be a restriction of your application
>>>> than a behavior bound at the interface level.  In fact, the latter
>>>> almost guarantees that MOAB will only support one part per
>>>> process, since that will cover 99% of the use cases.
>>>>
>>>> I think this goes back again to the runtime notion of a part being
>>>> confused with the unit of partitioning that's stored with /
>>>> associated to the mesh.  Maybe that's just a narrow view, though,
>>>> since I'm pretty sure Mark S. disagrees with that.
>>>>
>>>> - tim
>>>>
>>>> Mark Beall wrote:
>>>>> All,
>>>>> I was thinking over the call we had on Monday, specifically about
>>>>> what arguments were made against part=mesh instance. The only
>>>>> really compelling argument I recall (and sorry if I don't
>>>>> remember others, that's why I'm writing this email) was Tim's
>>>>> example of the overhead in partitioning a mesh into 100,000
>>>>> partitions with 8 elements each.
>>>>> Well, it kind of struck me that Tim's example, while relevant in
>>>>> terms of the percentage of overhead isn't really that relevant in
>>>>> terms of total memory. The initial mesh there would be 800,000
>>>>> elements, maybe a few hundred MB. Even with much more than 100%
>>>>> overhead, I could easily do that on my laptop. Given that I can
>>>>> buy a computer with 96 GB of memory today for about $8000 (192 GB
>>>>> for $17000) ( a Dell 7500 with 3rd party memory in case you're
>>>>> curious), you could add a couple zeros to the number of
>>>>> partitions for that mesh before it should become an issue for
>>>>> someone that will be running that simulation on a supercomputer
>>>>> costing a few hundred million dollars.
>>>>> What were the other compelling arguments against part=mesh
>>>>> instance?
>>>>> mark
>>>>
>>>> --
>>>> ================================================================
>>>> "You will keep in perfect peace him whose mind is
>>>> steadfast, because he trusts in you."               Isaiah 26:3
>>>>
>>>>            Tim Tautges            Argonne National Laboratory
>>>>        (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>>>>        phone: (608) 263-8485      1500 Engineering Dr.
>>>>          fax: (608) 263-4499      Madison, WI 53706
>>>>
>>
>> --
>> ================================================================
>> "You will keep in perfect peace him whose mind is
>>  steadfast, because he trusts in you."               Isaiah 26:3
>>
>>             Tim Tautges            Argonne National Laboratory
>>         (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>>         phone: (608) 263-8485      1500 Engineering Dr.
>>           fax: (608) 263-4499      Madison, WI 53706
>>
>
>