[Nek5000-users] scaling of nek5000 on a Cray

[nek5000-users at lists.mcs.anl.gov]

Hello Stefan,

Here is a (shortened) log file pertaining to a simulation on 4096 
processors of `our' Cray XT6m. The case is as follows:

* Nek5000, rev. 565
* 220248 elements in total, i.e. 54 elements/core
* Restart from a nearly converged initial condition

To your hints given below:
How do I enable Nek's internal MPI profiling?
We got a long list of runtime statistics and core-wise times; maybe the 
internal
profiling was actually active...
Could you give a brief explanation of the most important numbers, please?

How can I invoke the optimized MPI rank mapping?

Thank you in advance for some clarification!

Lars
(who collaborates with Johan)


nek5000-users at lists.mcs.anl.gov wrote:
> Hi Johan,
>
> we can definitely do better on Cray systems but I guess we need to do some fine tuning first.
> The BG architecture is quite different and you need much more grid points per core to scale on the Cray (4-8x more). 
>
> - Can you post your logfiles?
> - Did you try to enable Nek's internal MPI profiling where you spend most of the communication cost?
> - Did you try to use an optimized MPI rank mapping?
>
> Is it possible to get an account on that machine to do some experiments? 
>
>
> Cheers,
> Stefan
>
>
> ----- Original Message -----
> From: nek5000-users at lists.mcs.anl.gov
> To: nek5000-users at lists.mcs.anl.gov
> Sent: Tue, 19 Oct 2010 05:18:41 -0600 (GMT-06:00)
> Subject: [Nek5000-users] scaling of nek5000 on a Cray
>
> Hi nek5000-users & developers,
>
> I wonder if anyone has experience with the performance of nek5000 on a
> Cray ? In particular what scaling could one expect compared to e.g. a
> Blue Gene?
>
> We have a new Cray XT6m system, based on the AMD Opteron 12-core
> “Magny-Cours” <http://www.cray.com/Products/XT/Specifications.aspx> (2.1
> GHz) processors and the Cray SeaStar2 interconnect technology. It
> consists of 11040 compute cores and 10 service cores.
>
> The scaling is not too encouraging (see attached pdf-files). The
> attached files show three different runs, and the scaling is apparently
> highly dependent on where in the torus the job happens to be. This is
> maybe ok, but is the leveling off after 2048 cores consistent with what
> people observe on a Cray ?
>
> Thanks for your suggestions and experience!
>
> Best regards,
>
> Johan
>
>   
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cray_4096procs.out
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20101020/0434dded/attachment.ksh>