[Nek5000-users] Scalability

[nek5000-users at lists.mcs.anl.gov]

Shriram,

Total memory usage _per_processor_, scales as:

    memory/proc ~ [ C * lelt * lx1*ly1*lz1 +  c * lelg ] * (8 bytes/word)

where C is a moving target, but is somewhere between 200 and 400, and c ~ 2. 
(The code is a bit fat right now and needs a good clean-up, which is slated 
for next fall.)   Also, C can be influenced by some parameters in the
SIZEu file, namely, mxprev, lgmres, ldimt.

If your problem has nelt=120,000 elements and you're running on 1000
processors, you should set lelt=120 and lelg=120000.  This would give
you the smallest possible memory footprint for that case.   Note that
some machines, such as BG/L or BG/P have very little memory per MPI
process (e.g., 256 MB on BG/L and 512 MB on BG/P, of which only about
80% is available for the simulation).

Does this help?

Paul


On Tue, 2 Feb 2010, nek5000-users at lists.mcs.anl.gov wrote:

> Hi,
>
> When I am trying to run a big simulation (Nele > 120,000) , on certain
> clusters / supercomputers, Nek5000 crashes even though the physical memory
> used is very less. For example, on certain machines, with Nele = 120,000, we
> couldn't go more than lx1 of 7 while on different architecture we could go
> up to lx1 = 12. Is this issue concerned with compilers or the architecture
> of the machine ?
>
> Thanks
> Shriram Jagannathan
> Texas A&M University
>