[Nek5000-users] Running Simulations in Parallel with nekbmpi
nek5000-users at lists.mcs.anl.gov
nek5000-users at lists.mcs.anl.gov
Tue Oct 18 12:22:16 CDT 2011
How do you launch Nek in parallel? Can you try to run Nek using
mpiexec or mpirun w/o calling a script.
-Stefan
On 10/18/11, nek5000-users at lists.mcs.anl.gov
<nek5000-users at lists.mcs.anl.gov> wrote:
> Here it is:
>
> #!/bin/bash
> # Nek5000 build config file
> # (c) 2008,2009,2010 UCHICAGO ARGONNE, LLC
>
> # source path
> SOURCE_ROOT="$HOME/Codes/nek5_svn/trunk/nek"
>
> # Fortran compiler
> F77="mpif77"
>
> # C compiler
> CC="mpicc"
>
> # pre-processor symbol list
> # (set PPLIST=? to get a list of available symbols)
> #PPLIST="?"
>
> # plug-in list
> PLUGIN_LIST=""
>
>
> # OPTIONAL SETTINGS
> # -----------------
>
> # enable MPI (default true)
> #IFMPI="false"
>
> # auxilliary files to compile
> # NOTE: source files have to located in the same directory as makenek
> # a makefile_usr.inc has to be provided containing the build rules
> #USR="foo.o"
>
> # linking flags
> #USR_LFLAGS="-L/usr/lib -lfoo"
>
> # generic compiler flags
> #G="-g"
>
> # optimization flags
> #OPT_FLAGS_STD=""
> #OPT_FLAGS_MAG=""
>
> # enable AMG coarse grid solver (default XXT)
> #IFAMG="true"
> #IFAMG_DUMP="true"
>
> # CVODE path
> #CVODE_DIR=$HOME/cvode/lib
>
> # MOAB/iMESH path
> #MOAB_DIR="$HOME/moab"
>
>
> ###############################################################################
> # DONT'T TOUCH WHAT FOLLOWS !!!
> ###############################################################################
> # assign version tag
> mver=1
> # overwrite source path with optional 2nd argument
> if [ -d $2 ] && [ $# -eq 2 ]; then
> SOURCE_ROOT="$2"
> echo "change source code directory to: ", $SOURCE_ROOT
> fi
> # do some checks and create makefile
> source $SOURCE_ROOT/makenek.inc
> # compile
> make -j4 -f makefile 2>&1 | tee compiler.out
> exit 0
>
>
> On 18 October 2011 19:13, <nek5000-users at lists.mcs.anl.gov> wrote:
>
>> Please post your makenek.
>> -Stefan
>>
>> On 10/18/11, nek5000-users at lists.mcs.anl.gov
>> <nek5000-users at lists.mcs.anl.gov> wrote:
>> > Hi,
>> >
>> > Sorry for this very late response, it completely got out of my mind.
>> > Here
>> > are the different outputs I get for these commands:
>> >
>> > jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ ulimit -a
>> > core file size (blocks, -c) 0
>> > data seg size (kbytes, -d) unlimited
>> > scheduling priority (-e) 0
>> > file size (blocks, -f) unlimited
>> > pending signals (-i) 31533
>> > max locked memory (kbytes, -l) 64
>> > max memory size (kbytes, -m) unlimited
>> > open files (-n) 1024
>> > pipe size (512 bytes, -p) 8
>> > POSIX message queues (bytes, -q) 819200
>> > real-time priority (-r) 0
>> > stack size (kbytes, -s) 8192
>> > cpu time (seconds, -t) unlimited
>> > max user processes (-u) 31533
>> > virtual memory (kbytes, -v) unlimited
>> > file locks (-x) unlimited
>> >
>> > jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ size nek5000
>> > text data bss dec hex filename
>> > 2232834 3484 185835044 188071362 b35bdc2 nek5000
>> >
>> > jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ cat SIZE
>> > C Dimension file to be included
>> > C
>> > C HCUBE array dimensions
>> > C
>> > parameter (ldim=2)
>> > parameter (lx1=11,ly1=lx1,lz1=1,lelt=225,lelv=lelt)
>> > parameter (lxd=17,lyd=lxd,lzd=1)
>> > parameter (lelx=20,lely=60,lelz=1)
>> > c
>> > parameter (lzl=3 + 2*(ldim-3))
>> > c
>> > parameter (lx2=lx1-2)
>> > parameter (ly2=ly1-2)
>> > parameter (lz2=1)
>> > parameter (lx3=lx2)
>> > parameter (ly3=ly2)
>> > parameter (lz3=lz2)
>> > c
>> > c parameter (lpelv=lelv,lpelt=lelt,lpert=3) ! perturbation
>> > c parameter (lpx1=lx1,lpy1=ly1,lpz1=lz1) ! array sizes
>> > c parameter (lpx2=lx2,lpy2=ly2,lpz2=lz2)
>> > c
>> > parameter (lpelv=1,lpelt=1,lpert=1) ! perturbation
>> > parameter (lpx1=1,lpy1=1,lpz1=1) ! array sizes
>> > parameter (lpx2=1,lpy2=1,lpz2=1)
>> > c
>> > c
>> > c parameter (lbelv=lelv,lbelt=lelt) ! MHD
>> > c parameter (lbx1=lx1,lby1=ly1,lbz1=lz1) ! array sizes
>> > c parameter (lbx2=lx2,lby2=ly2,lbz2=lz2)
>> > c
>> > parameter (lbelv=1,lbelt=1) ! MHD
>> > parameter (lbx1=1,lby1=1,lbz1=1) ! array sizes
>> > parameter (lbx2=1,lby2=1,lbz2=1)
>> > c
>> > C LX1M=LX1 when there are moving meshes; =1 otherwise
>> > parameter (lx1m=lx1,ly1m=ly1,lz1m=lz1)
>> > parameter (ldimt= 4) ! 3 passive scalars + T
>> > parameter (ldimt1=ldimt+1)
>> > parameter (ldimt3=ldimt+3)
>> > parameter (lp = 8)
>> > parameter (lelg = 300)
>> > c
>> > c Note: In the new code, LELGEC should be about sqrt(LELG)
>> > c
>> > PARAMETER (LELGEC = 1)
>> > PARAMETER (LXYZ2 = 1)
>> > PARAMETER (LXZ21 = 1)
>> > c
>> > PARAMETER (LMAXV=LX1*LY1*LZ1*LELV)
>> > PARAMETER (LMAXT=LX1*LY1*LZ1*LELT)
>> > PARAMETER (LMAXP=LX2*LY2*LZ2*LELV)
>> > PARAMETER (LXZ=LX1*LZ1)
>> > PARAMETER (LORDER=3)
>> > PARAMETER (MAXOBJ=4,MAXMBR=LELT*6,lhis=100)
>> > C
>> > C Common Block Dimensions
>> > C
>> > PARAMETER (LCTMP0 =2*LX1*LY1*LZ1*LELT)
>> > PARAMETER (LCTMP1 =4*LX1*LY1*LZ1*LELT)
>> > C
>> > C The parameter LVEC controls whether an additional 42 field arrays
>> > C are required for Steady State Solutions. If you are not using
>> > C Steady State, it is recommended that LVEC=1.
>> > C
>> > PARAMETER (LVEC=1)
>> > C
>> > C Uzawa projection array dimensions
>> > C
>> > parameter (mxprev = 80)
>> > parameter (lgmres = 40)
>> > C
>> > C Split projection array dimensions
>> > C
>> > parameter(lmvec = 1)
>> > parameter(lsvec = 1)
>> > parameter(lstore=lmvec*lsvec)
>> > c
>> > c NONCONFORMING STUFF
>> > c
>> > parameter (maxmor = lelt)
>> > C
>> > C Array dimensions
>> > C
>> > COMMON/DIMN/NELV,NELT,NX1,NY1,NZ1,NX2,NY2,NZ2
>> > $,NX3,NY3,NZ3,NDIM,NFIELD,NPERT,NID
>> > $,NXD,NYD,NZD
>> >
>> > c automatically added by makenek
>> > parameter(lxo = lx1) ! max output grid size (lxo>=lx1)
>> >
>> > c automatically added by makenek
>> > parameter(lpart = 1 ) ! max number of particles
>> >
>> > c automatically added by makenek
>> > integer ax1,ay1,az1,ax2,ay2,az2
>> > parameter (ax1=lx1,ay1=ly1,az1=lz1,ax2=lx2,ay2=ly2,az2=lz2) !
>> running
>> > averages
>> >
>> > c automatically added by makenek
>> > parameter (lxs=1,lys=lxs,lzs=(lxs-1)*(ldim-2)+1) !New Pressure
>> > Preconditioner
>> >
>> > c automatically added by makenek
>> > parameter (lfdm=0) ! == 1 for fast diagonalization method
>> >
>> > Concerning Nek, yes I do get an output on the screen when lauching the
>> > calculation. Attached is the logfile for one example (2D cavity, only
>> with
>> > 100 elements though but the problem is the same as I increase the number
>> of
>> > elements).
>> >
>> > Cheers,
>> >
>> > On 18 October 2011 18:03, Jean-Christophe Loiseau
>> > <loiseau.jc at gmail.com>wrote:
>> >
>> >> Hi,
>> >>
>> >> Sorry for this very late response, it completely got out of my mind.
>> Here
>> >> are the different outputs I get for these commands:
>> >>
>> >> jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ ulimit -a
>> >> core file size (blocks, -c) 0
>> >> data seg size (kbytes, -d) unlimited
>> >> scheduling priority (-e) 0
>> >> file size (blocks, -f) unlimited
>> >> pending signals (-i) 31533
>> >> max locked memory (kbytes, -l) 64
>> >> max memory size (kbytes, -m) unlimited
>> >> open files (-n) 1024
>> >> pipe size (512 bytes, -p) 8
>> >> POSIX message queues (bytes, -q) 819200
>> >> real-time priority (-r) 0
>> >> stack size (kbytes, -s) 8192
>> >> cpu time (seconds, -t) unlimited
>> >> max user processes (-u) 31533
>> >> virtual memory (kbytes, -v) unlimited
>> >> file locks (-x) unlimited
>> >>
>> >> jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ size nek5000
>> >> text data bss dec hex filename
>> >> 2232834 3484 185835044 188071362 b35bdc2 nek5000
>> >>
>> >> jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ cat SIZE
>> >> C Dimension file to be included
>> >> C
>> >> C HCUBE array dimensions
>> >> C
>> >> parameter (ldim=2)
>> >> parameter (lx1=11,ly1=lx1,lz1=1,lelt=225,lelv=lelt)
>> >> parameter (lxd=17,lyd=lxd,lzd=1)
>> >> parameter (lelx=20,lely=60,lelz=1)
>> >> c
>> >> parameter (lzl=3 + 2*(ldim-3))
>> >> c
>> >> parameter (lx2=lx1-2)
>> >> parameter (ly2=ly1-2)
>> >> parameter (lz2=1)
>> >> parameter (lx3=lx2)
>> >> parameter (ly3=ly2)
>> >> parameter (lz3=lz2)
>> >> c
>> >> c parameter (lpelv=lelv,lpelt=lelt,lpert=3) ! perturbation
>> >> c parameter (lpx1=lx1,lpy1=ly1,lpz1=lz1) ! array sizes
>> >> c parameter (lpx2=lx2,lpy2=ly2,lpz2=lz2)
>> >> c
>> >> parameter (lpelv=1,lpelt=1,lpert=1) ! perturbation
>> >> parameter (lpx1=1,lpy1=1,lpz1=1) ! array sizes
>> >> parameter (lpx2=1,lpy2=1,lpz2=1)
>> >> c
>> >> c
>> >> c parameter (lbelv=lelv,lbelt=lelt) ! MHD
>> >> c parameter (lbx1=lx1,lby1=ly1,lbz1=lz1) ! array sizes
>> >> c parameter (lbx2=lx2,lby2=ly2,lbz2=lz2)
>> >> c
>> >> parameter (lbelv=1,lbelt=1) ! MHD
>> >> parameter (lbx1=1,lby1=1,lbz1=1) ! array sizes
>> >> parameter (lbx2=1,lby2=1,lbz2=1)
>> >> c
>> >> C LX1M=LX1 when there are moving meshes; =1 otherwise
>> >> parameter (lx1m=lx1,ly1m=ly1,lz1m=lz1)
>> >> parameter (ldimt= 4) ! 3 passive scalars +
>> >> T
>> >> parameter (ldimt1=ldimt+1)
>> >> parameter (ldimt3=ldimt+3)
>> >> parameter (lp = 8)
>> >> parameter (lelg = 300)
>> >> c
>> >> c Note: In the new code, LELGEC should be about sqrt(LELG)
>> >> c
>> >> PARAMETER (LELGEC = 1)
>> >> PARAMETER (LXYZ2 = 1)
>> >> PARAMETER (LXZ21 = 1)
>> >> c
>> >> PARAMETER (LMAXV=LX1*LY1*LZ1*LELV)
>> >> PARAMETER (LMAXT=LX1*LY1*LZ1*LELT)
>> >> PARAMETER (LMAXP=LX2*LY2*LZ2*LELV)
>> >> PARAMETER (LXZ=LX1*LZ1)
>> >> PARAMETER (LORDER=3)
>> >> PARAMETER (MAXOBJ=4,MAXMBR=LELT*6,lhis=100)
>> >> C
>> >> C Common Block Dimensions
>> >> C
>> >> PARAMETER (LCTMP0 =2*LX1*LY1*LZ1*LELT)
>> >> PARAMETER (LCTMP1 =4*LX1*LY1*LZ1*LELT)
>> >> C
>> >> C The parameter LVEC controls whether an additional 42 field arrays
>> >> C are required for Steady State Solutions. If you are not using
>> >> C Steady State, it is recommended that LVEC=1.
>> >> C
>> >> PARAMETER (LVEC=1)
>> >> C
>> >> C Uzawa projection array dimensions
>> >> C
>> >> parameter (mxprev = 80)
>> >> parameter (lgmres = 40)
>> >> C
>> >> C Split projection array dimensions
>> >> C
>> >> parameter(lmvec = 1)
>> >> parameter(lsvec = 1)
>> >> parameter(lstore=lmvec*lsvec)
>> >> c
>> >> c NONCONFORMING STUFF
>> >> c
>> >> parameter (maxmor = lelt)
>> >> C
>> >> C Array dimensions
>> >> C
>> >> COMMON/DIMN/NELV,NELT,NX1,NY1,NZ1,NX2,NY2,NZ2
>> >> $,NX3,NY3,NZ3,NDIM,NFIELD,NPERT,NID
>> >> $,NXD,NYD,NZD
>> >>
>> >> c automatically added by makenek
>> >> parameter(lxo = lx1) ! max output grid size (lxo>=lx1)
>> >>
>> >> c automatically added by makenek
>> >> parameter(lpart = 1 ) ! max number of particles
>> >>
>> >> c automatically added by makenek
>> >> integer ax1,ay1,az1,ax2,ay2,az2
>> >> parameter (ax1=lx1,ay1=ly1,az1=lz1,ax2=lx2,ay2=ly2,az2=lz2) !
>> >> running
>> >> averages
>> >>
>> >> c automatically added by makenek
>> >> parameter (lxs=1,lys=lxs,lzs=(lxs-1)*(ldim-2)+1) !New Pressure
>> >> Preconditioner
>> >>
>> >> c automatically added by makenek
>> >> parameter (lfdm=0) ! == 1 for fast diagonalization method
>> >>
>> >> Concerning Nek, yes I do get an output on the screen when lauching the
>> >> calculation. Attached is the logfile for one example (2D cavity, only
>> with
>> >> 100 elements though but the problem is the same as I increase the
>> >> number
>> >> of
>> >> elements).
>> >>
>> >> Cheers,
>> >>
>> >>
>> >> On 27 September 2011 17:40, <nek5000-users at lists.mcs.anl.gov> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Do you get any output printed to the screen when you launch Nek?
>> >>> Please provide the output of the following command:
>> >>> ulimit -a (in case you are using bash)
>> >>> size nek5000
>> >>> cat SIZE
>> >>>
>> >>> Can you post the command you are using to launch Nek in parallel?
>> >>>
>> >>> Cheers,
>> >>> Stefan
>> >>>
>> >>>
>> >>>
>> >>> On 9/27/11, nek5000-users at lists.mcs.anl.gov
>> >>> <nek5000-users at lists.mcs.anl.gov> wrote:
>> >>> > Hi Neks,
>> >>> >
>> >>> > I do get the same error when I try to run nek on my laptop. Did you
>> >>> > find
>> >>> any
>> >>> > way to solve the problem John?
>> >>> >
>> >>> > Regards,
>> >>> >
>> >>> > On 11 May 2011 21:25, <nek5000-users at lists.mcs.anl.gov> wrote:
>> >>> >
>> >>> >> Hi,
>> >>> >>
>> >>> >> I have lp=512 in SIZE, and I did compile with IFMPI=true. I
>> >>> >> recompiled
>> >>> >> yesterday because I thought this may be the case, but I was still
>> >>> getting
>> >>> >> the same error. When I first start a simulation and run top, it
>> shows
>> >>> >> four
>> >>> >> processors running nek5000, but this changes to one processor after
>> a
>> >>> few
>> >>> >> seconds.
>> >>> >>
>> >>> >> Thanks,
>> >>> >>
>> >>> >> John
>> >>> >>
>> >>> >> _______________________________________________
>> >>> >> Nek5000-users mailing list
>> >>> >> Nek5000-users at lists.mcs.anl.gov
>> >>> >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>> >>> >>
>> >>> >>
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Jean-Christophe
>> >>> >
>> >>> _______________________________________________
>> >>> Nek5000-users mailing list
>> >>> Nek5000-users at lists.mcs.anl.gov
>> >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Jean-Christophe
>> >>
>> >
>> >
>> >
>> > --
>> > Jean-Christophe
>> >
>> _______________________________________________
>> Nek5000-users mailing list
>> Nek5000-users at lists.mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>>
>
>
>
> --
> Jean-Christophe
>
More information about the Nek5000-users
mailing list