[Nek5000-users] scaling of nek5000 on a Cray

[nek5000-users at lists.mcs.anl.gov]

Hi Johan,

I've added some performance tests to the repo (v. 570)

There are a couple of calling options -

I would suggest, in usrdat, within  your .usr file to have


       iverbose = 1 ! fully verbose
       call platform_timer(iverbose)
       call exitti('timer done$',iverbose)

and run this with a reasonably large 3D case on your parallel
platform.

If you can send us the output of this run off list we'll
have a pretty good idea of what's going on.

Thanks!

Paul




On Wed, 20 Oct 2010, nek5000-users at lists.mcs.anl.gov wrote:

>
> I'm hoping to include our ping-pong and mxm timers in a future
> source release so that we can get a handle on the platform
> parameters via a couple of quick calls from userchk.  With luck,
> we'll have this in a few days.  --  Paul
>
>
>
> On Wed, 20 Oct 2010, nek5000-users at lists.mcs.anl.gov wrote:
>
>> Hi Johan,
>
> we can definitely do better on Cray systems but I guess we need to do some 
> fine tuning first.
> The BG architecture is quite different and you need much more grid points per 
> core to scale on the Cray (4-8x more).
>
> - Can you post your logfiles?
> - Did you try to enable Nek's internal MPI profiling where you spend most of 
> the communication cost?
> - Did you try to use an optimized MPI rank mapping?
>
> Is it possible to get an account on that machine to do some experiments?
>
>
> Cheers,
> Stefan
>
>
> ----- Original Message -----
> From: nek5000-users at lists.mcs.anl.gov
> To: nek5000-users at lists.mcs.anl.gov
> Sent: Tue, 19 Oct 2010 05:18:41 -0600 (GMT-06:00)
> Subject: [Nek5000-users] scaling of nek5000 on a Cray
>
> Hi nek5000-users & developers,
>
> I wonder if anyone has experience with the performance of nek5000 on a
> Cray ? In particular what scaling could one expect compared to e.g. a
> Blue Gene?
>
> We have a new Cray XT6m system, based on the AMD Opteron 12-core
> “Magny-Cours” <http://www.cray.com/Products/XT/Specifications.aspx> (2.1
> GHz) processors and the Cray SeaStar2 interconnect technology. It
> consists of 11040 compute cores and 10 service cores.
>
> The scaling is not too encouraging (see attached pdf-files). The
> attached files show three different runs, and the scaling is apparently
> highly dependent on where in the torus the job happens to be. This is
> maybe ok, but is the leveling off after 2048 cores consistent with what
> people observe on a Cray ?
>
> Thanks for your suggestions and experience!
>
> Best regards,
>
> Johan
>
> -- 
> Johan Malm
> Department of Mechanics, KTH
> SE-100 44, Stockholm, Sweden
> Phone: +46 8 7906876
> E-mail: johan at mech.kth.se
>
>
> _______________________________________________
> Nek5000-users mailing list
> Nek5000-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users