[Nek5000-users] scaling of nek5000 on a Cray

[nek5000-users at lists.mcs.anl.gov]

Thanks Paul,

Our system is currently down, but I'll do as you suggest as soon as the
system is up and running again.

Johan

nek5000-users at lists.mcs.anl.gov wrote:
>
> Hi Johan,
>
> I've added some performance tests to the repo (v. 570)
>
> There are a couple of calling options -
>
> I would suggest, in usrdat, within  your .usr file to have
>
>
>       iverbose = 1 ! fully verbose
>       call platform_timer(iverbose)
>       call exitti('timer done$',iverbose)
>
> and run this with a reasonably large 3D case on your parallel
> platform.
>
> If you can send us the output of this run off list we'll
> have a pretty good idea of what's going on.
>
> Thanks!
>
> Paul
>
>
>
>
> On Wed, 20 Oct 2010, nek5000-users at lists.mcs.anl.gov wrote:
>
>>
>> I'm hoping to include our ping-pong and mxm timers in a future
>> source release so that we can get a handle on the platform
>> parameters via a couple of quick calls from userchk.  With luck,
>> we'll have this in a few days.  --  Paul
>>
>>
>>
>> On Wed, 20 Oct 2010, nek5000-users at lists.mcs.anl.gov wrote:
>>
>>> Hi Johan,
>>
>> we can definitely do better on Cray systems but I guess we need to do
>> some fine tuning first.
>> The BG architecture is quite different and you need much more grid
>> points per core to scale on the Cray (4-8x more).
>>
>> - Can you post your logfiles?
>> - Did you try to enable Nek's internal MPI profiling where you spend
>> most of the communication cost?
>> - Did you try to use an optimized MPI rank mapping?
>>
>> Is it possible to get an account on that machine to do some experiments?
>>
>>
>> Cheers,
>> Stefan
>>
>>
>> ----- Original Message -----
>> From: nek5000-users at lists.mcs.anl.gov
>> To: nek5000-users at lists.mcs.anl.gov
>> Sent: Tue, 19 Oct 2010 05:18:41 -0600 (GMT-06:00)
>> Subject: [Nek5000-users] scaling of nek5000 on a Cray
>>
>> Hi nek5000-users & developers,
>>
>> I wonder if anyone has experience with the performance of nek5000 on a
>> Cray ? In particular what scaling could one expect compared to e.g. a
>> Blue Gene?
>>
>> We have a new Cray XT6m system, based on the AMD Opteron 12-core
>> “Magny-Cours” <http://www.cray.com/Products/XT/Specifications.aspx> (2.1
>> GHz) processors and the Cray SeaStar2 interconnect technology. It
>> consists of 11040 compute cores and 10 service cores.
>>
>> The scaling is not too encouraging (see attached pdf-files). The
>> attached files show three different runs, and the scaling is apparently
>> highly dependent on where in the torus the job happens to be. This is
>> maybe ok, but is the leveling off after 2048 cores consistent with what
>> people observe on a Cray ?
>>
>> Thanks for your suggestions and experience!
>>
>> Best regards,
>>
>> Johan
>>
>> -- 
>> Johan Malm
>> Department of Mechanics, KTH
>> SE-100 44, Stockholm, Sweden
>> Phone: +46 8 7906876
>> E-mail: johan at mech.kth.se
>>
>>
>> _______________________________________________
>> Nek5000-users mailing list
>> Nek5000-users at lists.mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
> ------------------------------------------------------------------------
>
> _______________________________________________
> Nek5000-users mailing list
> Nek5000-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>   


-- 
Johan Malm
Department of Mechanics, KTH
SE-100 44, Stockholm, Sweden
Phone: +46 8 7906876
E-mail: johan at mech.kth.se