[Nek5000-users] Help With Basic NEK Configuration

[nek5000-users at lists.mcs.anl.gov]

Hi Harish

I don't think the manual you are refering to is up-to-date. To get a
rough idea about your memory footprint use 'size nek5000' on the
command line. The size tool will report the total size of a static
data items. Note, the actual size of the memory footprint will be
different:

- Static data items which are not "touched" duing execution will not
be allocated, hence the acutal memory usage is lower. This is not true
on Blue Gene as the CNK-kernel does not use an over-optimistic memory
model.
- Dynamically allocated variables within Nek (gs-code) increase the
actual memory usage (I don't think more than 50MB but this depends on
the particular setup)
- Memory allocated by MPI (size depends on the MPI implemenation). On
Blue Gene this can be significant fraction considering the fact that a
communicator is ~6MB on 72 Racks. Note, Nek's AMG solver uses 50-80
communicators depending on the number of AMG levels. This can be
tricky but there are ways to reduce the memory footprint of a
communicator. Contact me again if you need more details on that.

The SIZE file is used to size all static data within Nek. Hence, all
numbers are considered as 'upper limits'. For more details check the
wiki:
http://nek5000.mcs.anl.gov/index.php/SIZEu

You can run Nek on any number of processors. However, the implemented
domain decompostion (RSB) works best if the number of processors is a
power of two. Here you'll find some parallel scaling data:
http://nek5000.mcs.anl.gov/index.php/Scaling

Cheers,
Stefan



On 8/5/11, nek5000-users at lists.mcs.anl.gov
<nek5000-users at lists.mcs.anl.gov> wrote:
> Hi,
>
> I am a new to NEK and I am trying to run a few examples for my OS scaling
> experiments. I went through the manual which is at:
> http://www.mcs.anl.gov/~fischer/new.pdf. In the chapter that speaks about
> a basic problem setup, there is a line in the 'Memory Requirements'
> section that says:
>
> Total Memory Required for simulation = E(N +1)^3 points × 400(wds/pt) ×
> 8bytes/wd
>
> I have been playing around with the SIZE file a little bit and adjusting,
> lx1, lelt, lxd, lp and lelg parameters. However I am unable to find where
> I can adjust 'E' from the equation. I understand that 'lelt' is maximum
> elements per processor and 'lelg' is maximum elements per simulation? Is
> 'lelg' same as 'E'? is 'lelt = lelg / nprocs' ? I assumed that 'E = lelg'
> and changed 'lelg' for 'timing' example to run it on a 4096 node
> configuration (1 rack of BG/P:Intrepid in VN mode). I keep getting the
> error:
>
> Too many processors ( 4096 ) for to few elements ( 512 )
>
> ... and the value 512 does not seem to change although I have changed
> 'lelg' and run 'makenek timing'. Is the value of 'E' picked up from the
> .re2 file? Would I have to generate it using 'genbox' after I change some
> other configuration?
>
> I understand that NEK has been scaled upto 40 racks of Intrepid at
> Argonne. I am also trying to run experiments of upto 32 racks if possible.
> But I want to get started with smaller runs. Is there a repository of
> configurations or tarred files where I could get larger, successfully run
> problem instances?
>
> Thanks.
>
> Regards,
> Harish Naik
> Predoctoral Appointee
> Mathematics & Computer Science Division
> Argonne National Laboratory
> P: (630) 252-2029
> W: http://www.mcs.anl.gov/~hnaik
> _______________________________________________
> Nek5000-users mailing list
> Nek5000-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>