[Nek5000-users] how to choose lx1

[nek5000-users at lists.mcs.anl.gov]

Hi Mehmet,

How long is nek at the gs_setup line? It can take several minutes for large
cases.

The last error you're having is because you've run out of memory. Remember
that the memory footprint on each processor scales as lx1^3 * lelt. With
certain compilers you can add the following option in makenek:

G="-mcmodel=medium"

This allows you compile larger arrays, and generally use more memory.

Best,
Justin

On Mon, Sep 24, 2012 at 4:17 PM, <nek5000-users at lists.mcs.anl.gov> wrote:

> Hi all,
>
> I'm trying to setup nek5000 with a high Rayleigh number problem.
> At first, I've used # of elements of 8192 with 64x16x8, in x,y,z
> respectively.
> And I set lex1 to 12 (thus lxd=15). When I compute
> (# of elements)x(order of polynomial+1)^3x400x8/(memory of a single proc.),
> I compute that aprox. 32 procs. will be fine to run the simulation.
>
> I compiled the model and that was fine. But I can only run the model with
> 32 procs.
> If I try to increase the number of procs. (48 or 64) it holds on the
> following line;
>
> gs_setup: 670615 unique labels shared
>    pairwise times (avg, min, max): 0.106644 0.089896 0.1268
>    crystal router                : 0.535782 0.507226 0.57275
>    used all_to_all method: pairwise
>    setupds time 2.8244E+01 seconds   4  8     5826591       43904
>  setup h1 coarse grid, nx_crs=           2
>  call usrsetvert
>  done :: usrsetvert
>
> gs_setup: 13483 unique labels shared
>    pairwise times (avg, min, max): 0.0878268 0.065232 0.1044
>
>
> I ran the model with 32 procs. for a while but I realized that it was too
> slow for my application.
> So I decided to setup another optimum version of it. What I read from the
> webpage is that I should choose lx1=7 or 9.
>
> So I increase my total number of elements to 112*28*14=43904.
> Now if I set lx1=9, lxd=15 and lelt=650 to run the model for 72 procs.
> It is still holding on the same location.
>
> If I set the lx1=8 and lxd=12,lelt=1220 then the model doesnot compile. It
> gives the following error;
>
> mpif90 -o nek5000 -fpp -DALLOW_USE_MPI -DALWAYS_USE_MPI obj/hc3d.o
> obj/drive.o obj/drive1.o obj/drive2.o obj/plan4.o obj/bdry.o obj/coef.o
> obj/conduct.o obj/connect1.o obj/connect2.o obj/dssum.o obj/edgec.o
> obj/eigsolv.o obj/gauss.o obj/genxyz.o obj/navier1.o obj/makeq.o
> obj/navier0.o obj/navier2.o obj/navier3.o obj/navier4.o obj/prepost.o
> obj/speclib.o obj/map2.o obj/turb.o obj/mvmesh.o obj/ic.o obj/ssolv.o
> obj/planx.o obj/math.o obj/mxm_wrapper.o obj/hmholtz.o obj/gfdm_par.o
> obj/gfdm_op.o obj/gfdm_solve.o obj/subs1.o obj/subs2.o obj/genbox.o
> obj/gmres.o obj/hsmg.o obj/convect.o obj/induct.o obj/perturb.o
> obj/navier5.o obj/navier6.o obj/navier7.o obj/navier8.o obj/fast3d.o
> obj/fasts.o obj/calcz.o obj/byte.o obj/chelpers.o obj/byte_mpi.o
> obj/postpro.o obj/cvode_driver.o obj/nek_comm.o obj/init_plugin.o
> obj/setprop.o obj/qthermal.o obj/cvode_aux.o obj/makeq_aux.o obj/papi.o
> obj/ssygv.o obj/dsygv.o obj/mxm_std.o obj/blas.o obj/comm_mpi.o
> obj/singlmesh.o obj/jl_gs.o obj/jl_sort.o obj/jl_sarray_transfer.o
> obj/jl_sarray_sort.o obj/jl_gs_local.o obj/jl_crystal.o obj/jl_comm.o
> obj/jl_tensor.o obj/jl_fail.o obj/jl_fcrystal.o obj/jl_findpts.o
> obj/jl_findpts_local.o obj/jl_obbox.o obj/jl_poly.o obj/jl_lob_bnd.o
> obj/jl_findpts_el_3.o obj/jl_findpts_el_2.o obj/jl_sparse_cholesky.o
> obj/jl_xxt.o obj/jl_fcrs.o
> obj/gmres.o: In function `gen_fast_g_':
> /home/ntnu/milicak/models/**nek5_svn/trunk/nek/gmres.f:(.**text+0xc4a9):
> relocation truncated to fit: R_X86_64_32S against symbol `fast1dsem_'
> defined in COMMON section in obj/fast3d.o
>
>
> I'm not quite sure what to do next.
> Thanks for any suggestions in advance,
> Mehmet
> ______________________________**_________________
> Nek5000-users mailing list
> Nek5000-users at lists.mcs.anl.**gov <Nek5000-users at lists.mcs.anl.gov>
> https://lists.mcs.anl.gov/**mailman/listinfo/nek5000-users<https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20120924/de005ee1/attachment.html>