mpi not working
Satish Balay
balay at mcs.anl.gov
Wed Jan 17 12:11:13 CST 2007
You've built PETSc with the following.
> > > > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0
However you are comparing simple MPI test with a different MPI [from
/opt/mpich/myrinet/intel]
> > > > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out
You are using different MPI imps for each of these cases - hence the
results are different.
I guess you should be using --with-mpi-dir=/opt/mpich/myrinet/intel
with PETSc configure.
If you still encounter problems - please send us the COMPLETE info wrt
the 2 tests.
i.e compile, run , output , with the location of compilers used [which mpif90]
Satish
On Wed, 17 Jan 2007, Ben Tay wrote:
> Hi,
>
> My school's server limit to minimum 4 processors. The output is
>
> Vector length 20
> Vector length 20
> Vector length 20 40 60
> All other values should be near zero
> VecScale 0
> VecCopy 0
> VecAXPY 0
> VecAYPX 0
> VecSwap 0
> VecSwap 0
> VecWAXPY 0
> VecPointwiseMult 0
> VecPointwiseDivide 0
> VecMAXPY 0 0 0
> Vector length 20 40 60
> All other values should be near zero
> VecScale 0
> VecCopy 0
> VecAXPY 0
> VecAYPX 0
> VecSwap 0
> VecSwap 0
> VecWAXPY 0
> VecPointwiseMult 0
> VecPointwiseDivide 0
> VecMAXPY 0 0 0
> Vector length 20
> Vector length 20 40 60
> All other values should be near zero
> Vector length 20
> Vector length 20 40 60
> All other values should be near zero
> VecScale 0
> VecCopy 0
> VecAXPY 0
> VecAYPX 0
> VecSwap 0
> VecSwap 0
> VecWAXPY 0
> VecPointwiseMult 0
> VecPointwiseDivide 0
> VecMAXPY 0 0 0
> VecScale 0
> VecCopy 0
> VecAXPY 0
> VecAYPX 0
> VecSwap 0
> VecSwap 0
> VecWAXPY 0
> VecPointwiseMult 0
> VecPointwiseDivide 0
> VecMAXPY 0 0 0
>
> So what's the verdict?
>
> Thank you.
>
>
> On 1/17/07, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >
> > Ben,
> >
> > I don't know what to say; what you report is inherently contradictory.
> > What happens when you run src/vec/vec/examples/tutorials/ex1.c on 2
> > processors?
> >
> > Barry
> >
> >
> > On Wed, 17 Jan 2007, Ben Tay wrote:
> >
> > > Thanks Shaman. But the problem is that I get
> > >
> > > 0,1
> > > 0,1
> > > 0,1
> > > 0,1
> > >
> > > instead of
> > >
> > > 0,4
> > > 1,4
> > > 2,4
> > > 3,4 which means there are 4 processors instead of the above, whereby it
> > > seems that 4 serial jobs are running.
> > >
> > > Barry:
> > >
> > > The script was given by my school when parallel jobs are to be
> > submitted. I
> > > use the same script when submitting pure mpi job and it works. On the
> > other
> > > hand, the PETSc parallel code ran successfully on another of my school's
> > > server. It was also submitted using a script, but a slightly different
> > one
> > > since it's another system. However, that server is very busy hence I
> > usually
> > > use the current server.
> > >
> > > Do you have other other solution? Or should I try other ways of
> > compilation?
> > > Btw, I am using ifc 7.0 and icc 7.0. The codes are written in fortran.
> > >
> > > Thank you.
> > >
> > >
> > > On 1/16/07, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > >
> > > >
> > > > Ben,
> > > >
> > > > You definitely have to submit a PETSc job just like
> > > > any other MPI job. So please try using the script.
> > > >
> > > > Barry
> > > >
> > > >
> > > > On Tue, 16 Jan 2007, Ben Tay wrote:
> > > >
> > > > > Hi Pan,
> > > > >
> > > > > I also got very big library files if I use PETSc with mpich2.
> > > > >
> > > > > Btw, I have tried several options but I still don't understand why I
> > > > can't
> > > > > get mpi to work with PETSc.
> > > > >
> > > > > The 4 processors are running together but each running its own code.
> > > > >
> > > > > I just use
> > > > >
> > > > >
> > > > > integer :: nprocs,rank,ierr
> > > > >
> > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr)
> > > > >
> > > > > call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
> > > > >
> > > > > call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr)
> > > > >
> > > > > print *, rank, nprocs
> > > > >
> > > > > call PetscFinalize(ierr)
> > > > >
> > > > >
> > > > >
> > > > > The answers I get is 0,1 repeated 4 times instead of 0,4 1,4 2,4
> > 3,4.
> > > > >
> > > > > I'm using my school's server's mpich and it work if I just compile
> > in
> > > > pure
> > > > > mpi.
> > > > >
> > > > > Btw, if I need to send the job to 4 processors, I need to use a
> > script
> > > > file:
> > > > >
> > > > > #BSUB -o std-output
> > > > > #BSUB -q linux_parallel_test
> > > > > #BSUB -n 4
> > > > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out
> > > > >
> > > > > I wonder if the problem lies here...
> > > > >
> > > > >
> > > > >
> > > > > Thank you.
> > > > >
> > > > >
> > > > >
> > > > > On 1/16/07, li pan <li76pan at yahoo.com> wrote:
> > > > > >
> > > > > > I did try to download and install petsc2.3.2, but end
> > > > > > up with error: mpich can not be download & installed,
> > > > > > please install mpich for windows manually.
> > > > > > In the homepage of mpich2, I didn't choose the version
> > > > > > for windows but the source code version. And compiled
> > > > > > it by myself. Then, I gave the --with-mpi-dir="install
> > > > > > dir". Petsc was configured, and now it's doing "make".
> > > > > > One interesting thing is, I installed petsc in linux
> > > > > > before. The mpi libraries were very large
> > > > > > (libmpich.a==60 mb). But this time in cygwin it was
> > > > > > only several mbs.
> > > > > >
> > > > > > best
> > > > > >
> > > > > > pan
> > > > > >
> > > > > >
> > > > > > --- Ben Tay <zonexo at gmail.com> wrote:
> > > > > >
> > > > > > > hi,
> > > > > > >
> > > > > > > i install PETSc using the following command:
> > > > > > >
> > > > > > > ./config/configure.py --with-vendor-compilers=intel
> > > > > > > --with-gnu-compilers=0
> > > > > > >
> > > > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32
> > > > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0
> > > > > > > --with-shared
> > > > > > >
> > > > > > > then i got:
> > > > > > >
> > > > > > > Compilers:
> > > > > > >
> > > > > > > C Compiler: /opt/mpich/intel/bin/mpicc
> > > > > > > -fPIC -g
> > > > > > > Fortran Compiler: /opt/mpich/intel/bin/mpif90
> > > > > > > -I. -fPIC -g -w90 -w
> > > > > > > Linkers:
> > > > > > > Shared linker: /opt/mpich/intel/bin/mpicc
> > > > > > > -shared -fPIC -g
> > > > > > > Dynamic linker: /opt/mpich/intel/bin/mpicc
> > > > > > > -shared -fPIC -g
> > > > > > > PETSc:
> > > > > > > PETSC_ARCH: linux-mpif90
> > > > > > > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8
> > > > > > > **
> > > > > > > ** Now build and test the libraries with "make all
> > > > > > > test"
> > > > > > > **
> > > > > > > Clanguage: C
> > > > > > > Scalar type:real
> > > > > > > MPI:
> > > > > > > Includes: ['/opt/mpich/intel/include']
> > > > > > > PETSc shared libraries: enabled
> > > > > > > PETSc dynamic libraries: disabled
> > > > > > > BLAS/LAPACK:
> > > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32
> > > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack
> > > > > > > -lmkl_ia32 -lguide
> > > > > > >
> > > > > > > i ran "make all test" and everything seems fine
> > > > > > >
> > > > > > > /opt/mpich/intel/bin/mpicc -c -fPIC -g
> > > > > > >
> > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc-
> > > > > > > 2.3.2-p8/bmake/linux-mpif90
> > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include
> > > > > > > -I/opt/mpich/intel/include
> > > > > > > -D__SDIR__="src/snes/examples/tutorials/" ex19.c
> > > > > > > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19
> > > > > > > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc-
> > > > > > > 2.3.2-p8/lib/linux-mpif90
> > > > > > >
> > > > > > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90
> > > > > > > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat
> > > > > > > -lpetscvec -lpetsc
> > > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32
> > > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack
> > > > > > > -lmkl_ia32 -lguide
> > > > > > > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib
> > > > > > > -Wl,-rpath,/opt/mpich/intel/lib
> > > > > > > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath
> > > > > > > -Wl,-rpath,-ldl -L-ldl -lmpich
> > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib
> > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib
> > > > > > > -L/opt/intel/compiler70/ia32/lib
> > > > > > > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib
> > > > > > > -limf -lirc -lcprts -lcxa
> > > > > > > -lunwind -ldl -lmpichf90 -lPEPCF90
> > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib
> > > > > > > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib
> > > > > > > -L/usr/lib -lintrins
> > > > > > > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\
> > > > > > > -ldl -lmpich
> > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib
> > > > > > > -L/opt/intel/compiler70/ia32/lib
> > > > > > > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts
> > > > > > > -lcxa -lunwind -ldl
> > > > > > > /bin/rm -f ex19.o
> > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run
> > > > > > > successfully with 1 MPI
> > > > > > > process
> > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run
> > > > > > > successfully with 2 MPI
> > > > > > > processes
> > > > > > > Fortran example src/snes/examples/tutorials/ex5f run
> > > > > > > successfully with 1 MPI
> > > > > > > process
> > > > > > > Completed test examples
> > > > > > >
> > > > > > > I then tried to run my own parallel code. It's a
> > > > > > > simple code which prints
> > > > > > > the rank of each processor.
> > > > > > >
> > > > > > > If I compile the code using just mpif90 test.F(using
> > > > > > > just mpif.h)
> > > > > > >
> > > > > > > I get 0,1,2,3 (4 processors).
> > > > > > >
> > > > > > > however, if i change the code to use petsc.h etc
> > > > > > > ie.
> > > > > > >
> > > > > > >
> > > > > > > program ns2d_c
> > > > > > >
> > > > > > > implicit none
> > > > > > >
> > > > > > >
> > > > > > > #include "include/finclude/petsc.h"
> > > > > > > #include "include/finclude/petscvec.h"
> > > > > > > #include "include/finclude/petscmat.h"
> > > > > > > #include "include/finclude/petscksp.h"
> > > > > > > #include "include/finclude/petscpc.h"
> > > > > > > #include "include/finclude/petscmat.h90"
> > > > > > >
> > > > > > > integer,parameter :: size_x=8,size_y=4
> > > > > > >
> > > > > > > integer ::
> > > > > > > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k
> > > > > > >
> > > > > > > PetscMPIInt nprocs,rank
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr)
> > > > > > >
> > > > > > > call
> > > > > > > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
> > > > > > >
> > > > > > > call
> > > > > > > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr)
> > > > > > >
> > > > > > > end program ns2d_c
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > i then rename the filename to ex2f.F and use "make
> > > > > > > ex2f"
> > > > > > >
> > > > > > > the result I get is something like 0,0,0,0.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Why is this so?
> > > > > > >
> > > > > > > Thank you.
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > >
> > ____________________________________________________________________________________
> > > > > > Never Miss an Email
> > > > > > Stay connected with Yahoo! Mail on your mobile. Get started!
> > > > > > http://mobile.yahoo.com/services?promote=mail
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> >
>
More information about the petsc-users
mailing list