[Nek5000-users] Good input dataset for NEK5000 runs

[nek5000-users at lists.mcs.anl.gov]

Hi Abhishek,

I would recommend turbChannel.

The example that is there has only 512 elements and thus probably wouldn't
be suitable at 256 cores (though it will certainly work up to an including
512 cores).

In general, there is a lower bound on the number of points per core, n/P,
required to in order to have enough work per core to offset the 
communication overhead.   For well-implemented domain-decomposition 
based strategies, this lower bound should be weakly dependent on the
code as there is a basically a certain amount of work to be done
(which grows as the domain volume - n/P) and a certain amount of
communication (which grows as the surface area ~ (n/P)^2/3).   If
communication is slow compared to the computation rate, then n/P must
be larger to give enough work.   This increase in n/P is generally
realized by decreasing the number of processors for the given problem (n).
The point where communication time equals computation time is a
reasonable estimate of the upper bound on P that can be effectively
used (though we would typically take P smaller than this figure).

The lower-bound n/P value for BG/P and BG/L architectures is n/P ~ 5000---10000
(points per core), at P=100,000.   The number is not stronly dependent on P.
(Again, this holds for any domain-decomposition strategy, not just Nek.)
We find, for example, parallel efficiency of 0.8 for P=131072 and n=10^9
(n/P=7300).    (See

     www.mcs.anl.gov/~fischer/sem1b

     www.mcs.anl.gov/~fischer/pubhtml

     Paul Fischer, James Lottes, David Pointer, and Andrew Siegel
     Petascale Algorithms for Reactor Hydrodynamics, J. Phys. Conf. Series (2008).

for discussion.)    NOTE that on a machine with a (relatively) slower
interconnect or faster clock, you will need larger n/P to get the same
parallel efficiency.

You can get a lot of platform information by calling "platform_timer(1)"
from the .usr file.    (This is time consuming, so best done as an independent
run.)   The platform_timer(iverbose) call provides estimates of flop rates
for matrix-matrix products relevant to nek and estimates of communication
latency and bandwidth.   From those, we can estimate the lower n/P bound.


Coming back to your question.  Let's assume you're running on a BG architecture
and you need n/P ~ 5000-10000.   If you run with ~5 elements per core and lx1=11,
you would have n/P = 5 x 10^3 = 5000.     If you want to run this at P=256, you
would need E = 256 x 5 elements = 1250 elements to realize reasonable efficiency.

In Nek, there are two ways to increase n:  increase number of elements (E) or
approximation order (N=lx1-1), for a total point count of n ~ EN^3.

For the turbChannel case, you can increase E by increasing the number of
elements in the box file, rerunning genbox followed by genmap to generate
a new .rea file, and then run the code.  You can also increase n by increasing
lx1.    The latter option is fine, but if you go too high you're probably not
in the regime where you would likely be doing production runs, though we've
certainly had cases at ANL and elsewhere where people have run with N=40 or more.

I hope this helps.

Paul









On Mon, 30 Apr 2012, nek5000-users at lists.mcs.anl.gov wrote:

> Hi,
>
> I am looking for a suitable input set for using NEK to compare some
> platforms (with diff interconnects). What is a good input set to do
> experiments up to 256 cores ? One from examples ? I came across papers
> where people have used G6a but I am unable to find that.
>
> Thanks,
> Abhishek
>
>
> On Mon, Apr 30, 2012 at 3:55 PM, Abhishek Gupta <gupta59 at illinois.edu>wrote:
>
>> Hi,
>>
>> I am looking for a suitable input set for using NEK to compare some
>> platforms (with diff interconnects). What is a good input set to do
>> experiments up to 256 cores ? One from examples ? I came across papers
>> where people have used G6a but I am unable to find that.
>>
>> Thanks,
>> Abhishek
>>
>>
>