itaps-parallel reasons against part=mesh instance

[Tim Tautges]

Some of the following argument depends on how partitions are handled.  However, assuming the partition is also 
associated with a mesh instance (with the partition in each mesh instance coordinated with those on other instances over 
the parallel job)... using one part per instance, and multiple instances on a given process, implies multiple threads of 
control, since many of the iMeshP functions are collective calls.  Single threads of control are far and away the most 
common mode for running codes at scale right now, and I assert will continue to be for some time (5-10 yrs).

Also, for the mode where an application is providing an iMeshP implementation on top of its data structure so it can use 
services implemented on iMeshP, I think the restriction of one part per instance means that these apps will always 
restrict themselves to one part per process.  I think your application is of this type.  So, I'd much rather have this 
be a restriction of your application than a behavior bound at the interface level.  In fact, the latter almost 
guarantees that MOAB will only support one part per process, since that will cover 99% of the use cases.

I think this goes back again to the runtime notion of a part being confused with the unit of partitioning that's stored 
with / associated to the mesh.  Maybe that's just a narrow view, though, since I'm pretty sure Mark S. disagrees with that.

- tim

Mark Beall wrote:
> All,
> 
> I was thinking over the call we had on Monday, specifically about what 
> arguments were made against part=mesh instance. The only really 
> compelling argument I recall (and sorry if I don't remember others, 
> that's why I'm writing this email) was Tim's example of the overhead in 
> partitioning a mesh into 100,000 partitions with 8 elements each.
> 
> Well, it kind of struck me that Tim's example, while relevant in terms 
> of the percentage of overhead isn't really that relevant in terms of 
> total memory. The initial mesh there would be 800,000 elements, maybe a 
> few hundred MB. Even with much more than 100% overhead, I could easily 
> do that on my laptop. Given that I can buy a computer with 96 GB of 
> memory today for about $8000 (192 GB for $17000) ( a Dell 7500 with 3rd 
> party memory in case you're curious), you could add a couple zeros to 
> the number of partitions for that mesh before it should become an issue 
> for someone that will be running that simulation on a supercomputer 
> costing a few hundred million dollars.
> 
> What were the other compelling arguments against part=mesh instance?
> 
> mark
> 
> 

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
          phone: (608) 263-8485      1500 Engineering Dr.
            fax: (608) 263-4499      Madison, WI 53706