[Nek5000-users] scaling of nek5000 on a Cray

[nek5000-users at lists.mcs.anl.gov]

Hi Johan,

we can definitely do better on Cray systems but I guess we need to do some fine tuning first.
The BG architecture is quite different and you need much more grid points per core to scale on the Cray (4-8x more). 

- Can you post your logfiles?
- Did you try to enable Nek's internal MPI profiling where you spend most of the communication cost?
- Did you try to use an optimized MPI rank mapping?

Is it possible to get an account on that machine to do some experiments? 


Cheers,
Stefan


----- Original Message -----
From: nek5000-users at lists.mcs.anl.gov
To: nek5000-users at lists.mcs.anl.gov
Sent: Tue, 19 Oct 2010 05:18:41 -0600 (GMT-06:00)
Subject: [Nek5000-users] scaling of nek5000 on a Cray

Hi nek5000-users & developers,

I wonder if anyone has experience with the performance of nek5000 on a
Cray ? In particular what scaling could one expect compared to e.g. a
Blue Gene?

We have a new Cray XT6m system, based on the AMD Opteron 12-core
“Magny-Cours” <http://www.cray.com/Products/XT/Specifications.aspx> (2.1
GHz) processors and the Cray SeaStar2 interconnect technology. It
consists of 11040 compute cores and 10 service cores.

The scaling is not too encouraging (see attached pdf-files). The
attached files show three different runs, and the scaling is apparently
highly dependent on where in the torus the job happens to be. This is
maybe ok, but is the leveling off after 2048 cores consistent with what
people observe on a Cray ?

Thanks for your suggestions and experience!

Best regards,

Johan

-- 
Johan Malm
Department of Mechanics, KTH
SE-100 44, Stockholm, Sweden
Phone: +46 8 7906876
E-mail: johan at mech.kth.se