[Nek5000-users] how to choose lx1

[nek5000-users at lists.mcs.anl.gov]

Hi all,

I'm trying to setup nek5000 with a high Rayleigh number problem.
At first, I've used # of elements of 8192 with 64x16x8, in x,y,z 
respectively.
And I set lex1 to 12 (thus lxd=15). When I compute
(# of elements)x(order of polynomial+1)^3x400x8/(memory of a single proc.),
I compute that aprox. 32 procs. will be fine to run the simulation.

I compiled the model and that was fine. But I can only run the model 
with 32 procs.
If I try to increase the number of procs. (48 or 64) it holds on the 
following line;

gs_setup: 670615 unique labels shared
    pairwise times (avg, min, max): 0.106644 0.089896 0.1268
    crystal router                : 0.535782 0.507226 0.57275
    used all_to_all method: pairwise
    setupds time 2.8244E+01 seconds   4  8     5826591       43904
  setup h1 coarse grid, nx_crs=           2
  call usrsetvert
  done :: usrsetvert

gs_setup: 13483 unique labels shared
    pairwise times (avg, min, max): 0.0878268 0.065232 0.1044


I ran the model with 32 procs. for a while but I realized that it was 
too slow for my application.
So I decided to setup another optimum version of it. What I read from 
the webpage is that I should choose lx1=7 or 9.

So I increase my total number of elements to 112*28*14=43904.
Now if I set lx1=9, lxd=15 and lelt=650 to run the model for 72 procs.
It is still holding on the same location.

If I set the lx1=8 and lxd=12,lelt=1220 then the model doesnot compile. 
It gives the following error;

mpif90 -o nek5000 -fpp -DALLOW_USE_MPI -DALWAYS_USE_MPI obj/hc3d.o 
obj/drive.o obj/drive1.o obj/drive2.o obj/plan4.o obj/bdry.o obj/coef.o 
obj/conduct.o obj/connect1.o obj/connect2.o obj/dssum.o obj/edgec.o 
obj/eigsolv.o obj/gauss.o obj/genxyz.o obj/navier1.o obj/makeq.o 
obj/navier0.o obj/navier2.o obj/navier3.o obj/navier4.o obj/prepost.o 
obj/speclib.o obj/map2.o obj/turb.o obj/mvmesh.o obj/ic.o obj/ssolv.o 
obj/planx.o obj/math.o obj/mxm_wrapper.o obj/hmholtz.o obj/gfdm_par.o 
obj/gfdm_op.o obj/gfdm_solve.o obj/subs1.o obj/subs2.o obj/genbox.o 
obj/gmres.o obj/hsmg.o obj/convect.o obj/induct.o obj/perturb.o 
obj/navier5.o obj/navier6.o obj/navier7.o obj/navier8.o obj/fast3d.o 
obj/fasts.o obj/calcz.o obj/byte.o obj/chelpers.o obj/byte_mpi.o 
obj/postpro.o obj/cvode_driver.o obj/nek_comm.o obj/init_plugin.o 
obj/setprop.o obj/qthermal.o obj/cvode_aux.o obj/makeq_aux.o obj/papi.o 
obj/ssygv.o obj/dsygv.o obj/mxm_std.o obj/blas.o obj/comm_mpi.o 
obj/singlmesh.o obj/jl_gs.o obj/jl_sort.o obj/jl_sarray_transfer.o 
obj/jl_sarray_sort.o obj/jl_gs_local.o obj/jl_crystal.o obj/jl_comm.o 
obj/jl_tensor.o obj/jl_fail.o obj/jl_fcrystal.o obj/jl_findpts.o 
obj/jl_findpts_local.o obj/jl_obbox.o obj/jl_poly.o obj/jl_lob_bnd.o 
obj/jl_findpts_el_3.o obj/jl_findpts_el_2.o obj/jl_sparse_cholesky.o 
obj/jl_xxt.o obj/jl_fcrs.o
obj/gmres.o: In function `gen_fast_g_':
/home/ntnu/milicak/models/nek5_svn/trunk/nek/gmres.f:(.text+0xc4a9): 
relocation truncated to fit: R_X86_64_32S against symbol `fast1dsem_' 
defined in COMMON section in obj/fast3d.o


I'm not quite sure what to do next.
Thanks for any suggestions in advance,
Mehmet