[Nek5000-users] Running Simulations in Parallel with nekbmpi
nek5000-users at lists.mcs.anl.gov
nek5000-users at lists.mcs.anl.gov
Tue Oct 18 14:30:04 CDT 2011
I tried:
1. mpiexec -n 2 ./nek5000
2. ./nekbmpi blah 2
and I get the same problem for both. There is no big deal if it doesn't work
on parallel since it is on my laptop, but it would be cool if it worked :)
I'll continue to look for the problem. It might perhaps come from my mpich
implementation. I'll try to run some other parallel codes and get back to
you after.
On 18 October 2011 19:22, <nek5000-users at lists.mcs.anl.gov> wrote:
> How do you launch Nek in parallel? Can you try to run Nek using
> mpiexec or mpirun w/o calling a script.
> -Stefan
>
> On 10/18/11, nek5000-users at lists.mcs.anl.gov
> <nek5000-users at lists.mcs.anl.gov> wrote:
> > Here it is:
> >
> > #!/bin/bash
> > # Nek5000 build config file
> > # (c) 2008,2009,2010 UCHICAGO ARGONNE, LLC
> >
> > # source path
> > SOURCE_ROOT="$HOME/Codes/nek5_svn/trunk/nek"
> >
> > # Fortran compiler
> > F77="mpif77"
> >
> > # C compiler
> > CC="mpicc"
> >
> > # pre-processor symbol list
> > # (set PPLIST=? to get a list of available symbols)
> > #PPLIST="?"
> >
> > # plug-in list
> > PLUGIN_LIST=""
> >
> >
> > # OPTIONAL SETTINGS
> > # -----------------
> >
> > # enable MPI (default true)
> > #IFMPI="false"
> >
> > # auxilliary files to compile
> > # NOTE: source files have to located in the same directory as makenek
> > # a makefile_usr.inc has to be provided containing the build rules
> > #USR="foo.o"
> >
> > # linking flags
> > #USR_LFLAGS="-L/usr/lib -lfoo"
> >
> > # generic compiler flags
> > #G="-g"
> >
> > # optimization flags
> > #OPT_FLAGS_STD=""
> > #OPT_FLAGS_MAG=""
> >
> > # enable AMG coarse grid solver (default XXT)
> > #IFAMG="true"
> > #IFAMG_DUMP="true"
> >
> > # CVODE path
> > #CVODE_DIR=$HOME/cvode/lib
> >
> > # MOAB/iMESH path
> > #MOAB_DIR="$HOME/moab"
> >
> >
> >
> ###############################################################################
> > # DONT'T TOUCH WHAT FOLLOWS !!!
> >
> ###############################################################################
> > # assign version tag
> > mver=1
> > # overwrite source path with optional 2nd argument
> > if [ -d $2 ] && [ $# -eq 2 ]; then
> > SOURCE_ROOT="$2"
> > echo "change source code directory to: ", $SOURCE_ROOT
> > fi
> > # do some checks and create makefile
> > source $SOURCE_ROOT/makenek.inc
> > # compile
> > make -j4 -f makefile 2>&1 | tee compiler.out
> > exit 0
> >
> >
> > On 18 October 2011 19:13, <nek5000-users at lists.mcs.anl.gov> wrote:
> >
> >> Please post your makenek.
> >> -Stefan
> >>
> >> On 10/18/11, nek5000-users at lists.mcs.anl.gov
> >> <nek5000-users at lists.mcs.anl.gov> wrote:
> >> > Hi,
> >> >
> >> > Sorry for this very late response, it completely got out of my mind.
> >> > Here
> >> > are the different outputs I get for these commands:
> >> >
> >> > jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ ulimit -a
> >> > core file size (blocks, -c) 0
> >> > data seg size (kbytes, -d) unlimited
> >> > scheduling priority (-e) 0
> >> > file size (blocks, -f) unlimited
> >> > pending signals (-i) 31533
> >> > max locked memory (kbytes, -l) 64
> >> > max memory size (kbytes, -m) unlimited
> >> > open files (-n) 1024
> >> > pipe size (512 bytes, -p) 8
> >> > POSIX message queues (bytes, -q) 819200
> >> > real-time priority (-r) 0
> >> > stack size (kbytes, -s) 8192
> >> > cpu time (seconds, -t) unlimited
> >> > max user processes (-u) 31533
> >> > virtual memory (kbytes, -v) unlimited
> >> > file locks (-x) unlimited
> >> >
> >> > jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ size nek5000
> >> > text data bss dec hex filename
> >> > 2232834 3484 185835044 188071362 b35bdc2 nek5000
> >> >
> >> > jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ cat SIZE
> >> > C Dimension file to be included
> >> > C
> >> > C HCUBE array dimensions
> >> > C
> >> > parameter (ldim=2)
> >> > parameter (lx1=11,ly1=lx1,lz1=1,lelt=225,lelv=lelt)
> >> > parameter (lxd=17,lyd=lxd,lzd=1)
> >> > parameter (lelx=20,lely=60,lelz=1)
> >> > c
> >> > parameter (lzl=3 + 2*(ldim-3))
> >> > c
> >> > parameter (lx2=lx1-2)
> >> > parameter (ly2=ly1-2)
> >> > parameter (lz2=1)
> >> > parameter (lx3=lx2)
> >> > parameter (ly3=ly2)
> >> > parameter (lz3=lz2)
> >> > c
> >> > c parameter (lpelv=lelv,lpelt=lelt,lpert=3) ! perturbation
> >> > c parameter (lpx1=lx1,lpy1=ly1,lpz1=lz1) ! array sizes
> >> > c parameter (lpx2=lx2,lpy2=ly2,lpz2=lz2)
> >> > c
> >> > parameter (lpelv=1,lpelt=1,lpert=1) ! perturbation
> >> > parameter (lpx1=1,lpy1=1,lpz1=1) ! array sizes
> >> > parameter (lpx2=1,lpy2=1,lpz2=1)
> >> > c
> >> > c
> >> > c parameter (lbelv=lelv,lbelt=lelt) ! MHD
> >> > c parameter (lbx1=lx1,lby1=ly1,lbz1=lz1) ! array sizes
> >> > c parameter (lbx2=lx2,lby2=ly2,lbz2=lz2)
> >> > c
> >> > parameter (lbelv=1,lbelt=1) ! MHD
> >> > parameter (lbx1=1,lby1=1,lbz1=1) ! array sizes
> >> > parameter (lbx2=1,lby2=1,lbz2=1)
> >> > c
> >> > C LX1M=LX1 when there are moving meshes; =1 otherwise
> >> > parameter (lx1m=lx1,ly1m=ly1,lz1m=lz1)
> >> > parameter (ldimt= 4) ! 3 passive scalars +
> T
> >> > parameter (ldimt1=ldimt+1)
> >> > parameter (ldimt3=ldimt+3)
> >> > parameter (lp = 8)
> >> > parameter (lelg = 300)
> >> > c
> >> > c Note: In the new code, LELGEC should be about sqrt(LELG)
> >> > c
> >> > PARAMETER (LELGEC = 1)
> >> > PARAMETER (LXYZ2 = 1)
> >> > PARAMETER (LXZ21 = 1)
> >> > c
> >> > PARAMETER (LMAXV=LX1*LY1*LZ1*LELV)
> >> > PARAMETER (LMAXT=LX1*LY1*LZ1*LELT)
> >> > PARAMETER (LMAXP=LX2*LY2*LZ2*LELV)
> >> > PARAMETER (LXZ=LX1*LZ1)
> >> > PARAMETER (LORDER=3)
> >> > PARAMETER (MAXOBJ=4,MAXMBR=LELT*6,lhis=100)
> >> > C
> >> > C Common Block Dimensions
> >> > C
> >> > PARAMETER (LCTMP0 =2*LX1*LY1*LZ1*LELT)
> >> > PARAMETER (LCTMP1 =4*LX1*LY1*LZ1*LELT)
> >> > C
> >> > C The parameter LVEC controls whether an additional 42 field
> arrays
> >> > C are required for Steady State Solutions. If you are not using
> >> > C Steady State, it is recommended that LVEC=1.
> >> > C
> >> > PARAMETER (LVEC=1)
> >> > C
> >> > C Uzawa projection array dimensions
> >> > C
> >> > parameter (mxprev = 80)
> >> > parameter (lgmres = 40)
> >> > C
> >> > C Split projection array dimensions
> >> > C
> >> > parameter(lmvec = 1)
> >> > parameter(lsvec = 1)
> >> > parameter(lstore=lmvec*lsvec)
> >> > c
> >> > c NONCONFORMING STUFF
> >> > c
> >> > parameter (maxmor = lelt)
> >> > C
> >> > C Array dimensions
> >> > C
> >> > COMMON/DIMN/NELV,NELT,NX1,NY1,NZ1,NX2,NY2,NZ2
> >> > $,NX3,NY3,NZ3,NDIM,NFIELD,NPERT,NID
> >> > $,NXD,NYD,NZD
> >> >
> >> > c automatically added by makenek
> >> > parameter(lxo = lx1) ! max output grid size (lxo>=lx1)
> >> >
> >> > c automatically added by makenek
> >> > parameter(lpart = 1 ) ! max number of particles
> >> >
> >> > c automatically added by makenek
> >> > integer ax1,ay1,az1,ax2,ay2,az2
> >> > parameter (ax1=lx1,ay1=ly1,az1=lz1,ax2=lx2,ay2=ly2,az2=lz2) !
> >> running
> >> > averages
> >> >
> >> > c automatically added by makenek
> >> > parameter (lxs=1,lys=lxs,lzs=(lxs-1)*(ldim-2)+1) !New Pressure
> >> > Preconditioner
> >> >
> >> > c automatically added by makenek
> >> > parameter (lfdm=0) ! == 1 for fast diagonalization method
> >> >
> >> > Concerning Nek, yes I do get an output on the screen when lauching the
> >> > calculation. Attached is the logfile for one example (2D cavity, only
> >> with
> >> > 100 elements though but the problem is the same as I increase the
> number
> >> of
> >> > elements).
> >> >
> >> > Cheers,
> >> >
> >> > On 18 October 2011 18:03, Jean-Christophe Loiseau
> >> > <loiseau.jc at gmail.com>wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> Sorry for this very late response, it completely got out of my mind.
> >> Here
> >> >> are the different outputs I get for these commands:
> >> >>
> >> >> jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ ulimit -a
> >> >> core file size (blocks, -c) 0
> >> >> data seg size (kbytes, -d) unlimited
> >> >> scheduling priority (-e) 0
> >> >> file size (blocks, -f) unlimited
> >> >> pending signals (-i) 31533
> >> >> max locked memory (kbytes, -l) 64
> >> >> max memory size (kbytes, -m) unlimited
> >> >> open files (-n) 1024
> >> >> pipe size (512 bytes, -p) 8
> >> >> POSIX message queues (bytes, -q) 819200
> >> >> real-time priority (-r) 0
> >> >> stack size (kbytes, -s) 8192
> >> >> cpu time (seconds, -t) unlimited
> >> >> max user processes (-u) 31533
> >> >> virtual memory (kbytes, -v) unlimited
> >> >> file locks (-x) unlimited
> >> >>
> >> >> jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ size nek5000
> >> >> text data bss dec hex filename
> >> >> 2232834 3484 185835044 188071362 b35bdc2 nek5000
> >> >>
> >> >> jean-christophe at dyf-corto:~/Calculs/Cavite2D/BaseFlow$ cat SIZE
> >> >> C Dimension file to be included
> >> >> C
> >> >> C HCUBE array dimensions
> >> >> C
> >> >> parameter (ldim=2)
> >> >> parameter (lx1=11,ly1=lx1,lz1=1,lelt=225,lelv=lelt)
> >> >> parameter (lxd=17,lyd=lxd,lzd=1)
> >> >> parameter (lelx=20,lely=60,lelz=1)
> >> >> c
> >> >> parameter (lzl=3 + 2*(ldim-3))
> >> >> c
> >> >> parameter (lx2=lx1-2)
> >> >> parameter (ly2=ly1-2)
> >> >> parameter (lz2=1)
> >> >> parameter (lx3=lx2)
> >> >> parameter (ly3=ly2)
> >> >> parameter (lz3=lz2)
> >> >> c
> >> >> c parameter (lpelv=lelv,lpelt=lelt,lpert=3) ! perturbation
> >> >> c parameter (lpx1=lx1,lpy1=ly1,lpz1=lz1) ! array sizes
> >> >> c parameter (lpx2=lx2,lpy2=ly2,lpz2=lz2)
> >> >> c
> >> >> parameter (lpelv=1,lpelt=1,lpert=1) ! perturbation
> >> >> parameter (lpx1=1,lpy1=1,lpz1=1) ! array sizes
> >> >> parameter (lpx2=1,lpy2=1,lpz2=1)
> >> >> c
> >> >> c
> >> >> c parameter (lbelv=lelv,lbelt=lelt) ! MHD
> >> >> c parameter (lbx1=lx1,lby1=ly1,lbz1=lz1) ! array sizes
> >> >> c parameter (lbx2=lx2,lby2=ly2,lbz2=lz2)
> >> >> c
> >> >> parameter (lbelv=1,lbelt=1) ! MHD
> >> >> parameter (lbx1=1,lby1=1,lbz1=1) ! array sizes
> >> >> parameter (lbx2=1,lby2=1,lbz2=1)
> >> >> c
> >> >> C LX1M=LX1 when there are moving meshes; =1 otherwise
> >> >> parameter (lx1m=lx1,ly1m=ly1,lz1m=lz1)
> >> >> parameter (ldimt= 4) ! 3 passive scalars
> +
> >> >> T
> >> >> parameter (ldimt1=ldimt+1)
> >> >> parameter (ldimt3=ldimt+3)
> >> >> parameter (lp = 8)
> >> >> parameter (lelg = 300)
> >> >> c
> >> >> c Note: In the new code, LELGEC should be about sqrt(LELG)
> >> >> c
> >> >> PARAMETER (LELGEC = 1)
> >> >> PARAMETER (LXYZ2 = 1)
> >> >> PARAMETER (LXZ21 = 1)
> >> >> c
> >> >> PARAMETER (LMAXV=LX1*LY1*LZ1*LELV)
> >> >> PARAMETER (LMAXT=LX1*LY1*LZ1*LELT)
> >> >> PARAMETER (LMAXP=LX2*LY2*LZ2*LELV)
> >> >> PARAMETER (LXZ=LX1*LZ1)
> >> >> PARAMETER (LORDER=3)
> >> >> PARAMETER (MAXOBJ=4,MAXMBR=LELT*6,lhis=100)
> >> >> C
> >> >> C Common Block Dimensions
> >> >> C
> >> >> PARAMETER (LCTMP0 =2*LX1*LY1*LZ1*LELT)
> >> >> PARAMETER (LCTMP1 =4*LX1*LY1*LZ1*LELT)
> >> >> C
> >> >> C The parameter LVEC controls whether an additional 42 field
> arrays
> >> >> C are required for Steady State Solutions. If you are not using
> >> >> C Steady State, it is recommended that LVEC=1.
> >> >> C
> >> >> PARAMETER (LVEC=1)
> >> >> C
> >> >> C Uzawa projection array dimensions
> >> >> C
> >> >> parameter (mxprev = 80)
> >> >> parameter (lgmres = 40)
> >> >> C
> >> >> C Split projection array dimensions
> >> >> C
> >> >> parameter(lmvec = 1)
> >> >> parameter(lsvec = 1)
> >> >> parameter(lstore=lmvec*lsvec)
> >> >> c
> >> >> c NONCONFORMING STUFF
> >> >> c
> >> >> parameter (maxmor = lelt)
> >> >> C
> >> >> C Array dimensions
> >> >> C
> >> >> COMMON/DIMN/NELV,NELT,NX1,NY1,NZ1,NX2,NY2,NZ2
> >> >> $,NX3,NY3,NZ3,NDIM,NFIELD,NPERT,NID
> >> >> $,NXD,NYD,NZD
> >> >>
> >> >> c automatically added by makenek
> >> >> parameter(lxo = lx1) ! max output grid size (lxo>=lx1)
> >> >>
> >> >> c automatically added by makenek
> >> >> parameter(lpart = 1 ) ! max number of particles
> >> >>
> >> >> c automatically added by makenek
> >> >> integer ax1,ay1,az1,ax2,ay2,az2
> >> >> parameter (ax1=lx1,ay1=ly1,az1=lz1,ax2=lx2,ay2=ly2,az2=lz2) !
> >> >> running
> >> >> averages
> >> >>
> >> >> c automatically added by makenek
> >> >> parameter (lxs=1,lys=lxs,lzs=(lxs-1)*(ldim-2)+1) !New Pressure
> >> >> Preconditioner
> >> >>
> >> >> c automatically added by makenek
> >> >> parameter (lfdm=0) ! == 1 for fast diagonalization method
> >> >>
> >> >> Concerning Nek, yes I do get an output on the screen when lauching
> the
> >> >> calculation. Attached is the logfile for one example (2D cavity, only
> >> with
> >> >> 100 elements though but the problem is the same as I increase the
> >> >> number
> >> >> of
> >> >> elements).
> >> >>
> >> >> Cheers,
> >> >>
> >> >>
> >> >> On 27 September 2011 17:40, <nek5000-users at lists.mcs.anl.gov> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> Do you get any output printed to the screen when you launch Nek?
> >> >>> Please provide the output of the following command:
> >> >>> ulimit -a (in case you are using bash)
> >> >>> size nek5000
> >> >>> cat SIZE
> >> >>>
> >> >>> Can you post the command you are using to launch Nek in parallel?
> >> >>>
> >> >>> Cheers,
> >> >>> Stefan
> >> >>>
> >> >>>
> >> >>>
> >> >>> On 9/27/11, nek5000-users at lists.mcs.anl.gov
> >> >>> <nek5000-users at lists.mcs.anl.gov> wrote:
> >> >>> > Hi Neks,
> >> >>> >
> >> >>> > I do get the same error when I try to run nek on my laptop. Did
> you
> >> >>> > find
> >> >>> any
> >> >>> > way to solve the problem John?
> >> >>> >
> >> >>> > Regards,
> >> >>> >
> >> >>> > On 11 May 2011 21:25, <nek5000-users at lists.mcs.anl.gov> wrote:
> >> >>> >
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> I have lp=512 in SIZE, and I did compile with IFMPI=true. I
> >> >>> >> recompiled
> >> >>> >> yesterday because I thought this may be the case, but I was still
> >> >>> getting
> >> >>> >> the same error. When I first start a simulation and run top, it
> >> shows
> >> >>> >> four
> >> >>> >> processors running nek5000, but this changes to one processor
> after
> >> a
> >> >>> few
> >> >>> >> seconds.
> >> >>> >>
> >> >>> >> Thanks,
> >> >>> >>
> >> >>> >> John
> >> >>> >>
> >> >>> >> _______________________________________________
> >> >>> >> Nek5000-users mailing list
> >> >>> >> Nek5000-users at lists.mcs.anl.gov
> >> >>> >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
> >> >>> >>
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > Jean-Christophe
> >> >>> >
> >> >>> _______________________________________________
> >> >>> Nek5000-users mailing list
> >> >>> Nek5000-users at lists.mcs.anl.gov
> >> >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Jean-Christophe
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Jean-Christophe
> >> >
> >> _______________________________________________
> >> Nek5000-users mailing list
> >> Nek5000-users at lists.mcs.anl.gov
> >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
> >>
> >
> >
> >
> > --
> > Jean-Christophe
> >
> _______________________________________________
> Nek5000-users mailing list
> Nek5000-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>
--
Jean-Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20111018/16e323cc/attachment.html>
More information about the Nek5000-users
mailing list