understanding the output from -info
Barry Smith
bsmith at mcs.anl.gov
Fri Feb 9 19:27:33 CST 2007
Ben,
1)
> >
> >
> > ##########################################################
> > # #
> > # WARNING!!! #
> > # #
> > # This code was compiled with a debugging option, #
> > # To get timing results run config/configure.py #
> > # using --with-debugging=no, the performance will #
> > # be generally two or three times faster. #
> > # #
> > ##########################################################
2) In general to get any decent parallel performance you need to have at
least 10,000 unknowns per process.
3) It is important that each proces have roughly the same number of nonzeros
in the matrix.
> > Event Count Time (sec)
> > Flops/sec --- Global --- --- Stage --- Total
> > Max Ratio Max Ratio Max Ratio Mess Avg len
> > MatSolve 3967 1.0 2.5914e+00 1.9 7.99e+07 1.9 0.0e+00 0.0e+00
^^^^^^
One process is taking 1.9 times for the matsolves then the fastest one. Since the
MatSolves are not parallel this likely means that the "slow process" has much more
nonzeros thant the "fast process"
Barry
On Fri, 9 Feb 2007, Matthew Knepley wrote:
> 1) These MFlop rates are terrible. It seems like your problem is way too
> small.
>
> 2) The load balance is not good.
>
> Matt
>
> On 2/9/07, Ben Tay <zonexo at gmail.com> wrote:
> >
> > Ya, that's the mistake. I changed part of the code resulting in
> > PetscFinalize not being called.
> >
> > Here's the output:
> >
> >
> > ---------------------------------------------- PETSc Performance Summary:
> > ----------------------------------------------
> >
> > /home/enduser/g0306332/ns2d/a.out on a linux-mpi named
> > atlas00.nus.edu.sgwith 4 processors, by g0306332 Sat Feb 10 08:32:08 2007
> > Using Petsc Release Version 2.3.2, Patch 8, Tue Jan 2 14:33:59 PST 2007
> > HG revision: ebeddcedcc065e32fc252af32cf1d01ed4fc7a80
> >
> > Max Max/Min Avg Total
> > Time (sec): 2.826e+02 2.08192 1.725e+02
> > Objects: 1.110e+02 1.00000 1.110e+02
> > Flops: 6.282e+08 1.00736 6.267e+08 2.507e+09
> > Flops/sec: 4.624e+06 2.08008 4.015e+06 1.606e+07
> > Memory: 1.411e+07 1.01142 5.610e+07
> > MPI Messages: 8.287e+03 1.90156 6.322e+03 2.529e+04
> > MPI Message Lengths: 6.707e+07 1.11755 1.005e+04 2.542e+08
> > MPI Reductions: 3.112e+03 1.00000
> >
> > Flop counting convention: 1 flop = 1 real number operation of type
> > (multiply/divide/add/subtract)
> > e.g., VecAXPY() for real vectors of length N
> > --> 2N flops
> > and VecAXPY() for complex vectors of length N
> > --> 8N flops
> >
> > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages
> > --- -- Message Lengths -- -- Reductions --
> > Avg %Total Avg %Total counts
> > %Total Avg %Total counts %Total
> > 0: Main Stage: 1.7247e+02 100.0% 2.5069e+09 100.0% 2.529e+04
> > 100.0% 1.005e+04 100.0% 1.245e+04 100.0%
> >
> >
> >
> > ------------------------------------------------------------------------------------------------------------------------
> > See the 'Profiling' chapter of the users' manual for details on
> > interpreting output.
> > Phase summary info:
> > Count: number of times phase was executed
> > Time and Flops/sec: Max - maximum over all processors
> > Ratio - ratio of maximum to minimum over all
> > processors
> > Mess: number of messages sent
> > Avg. len: average message length
> > Reduct: number of global reductions
> > Global: entire computation
> > Stage: stages of a computation. Set stages with PetscLogStagePush() and
> > PetscLogStagePop().
> > %T - percent time in this phase %F - percent flops in this
> > phase
> > %M - percent messages in this phase %L - percent message lengths
> > in this phase
> > %R - percent reductions in this phase
> > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time
> > over all processors)
> >
> >
> > ------------------------------------------------------------------------------------------------------------------------
> >
> >
> > ##########################################################
> > # #
> > # WARNING!!! #
> > # #
> > # This code was compiled with a debugging option, #
> > # To get timing results run config/configure.py #
> > # using --with-debugging=no, the performance will #
> > # be generally two or three times faster. #
> > # #
> > ##########################################################
> >
> >
> >
> >
> > ##########################################################
> >
> >
> >
> > ##########################################################
> > # #
> > # WARNING!!! #
> > # #
> > # This code was run without the PreLoadBegin() #
> > # macros. To get timing results we always recommend #
> > # preloading. otherwise timing numbers may be #
> > # meaningless. #
> > ##########################################################
> >
> >
> > Event Count Time (sec)
> > Flops/sec --- Global --- --- Stage --- Total
> > Max Ratio Max Ratio Max Ratio Mess Avg len
> > Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
> >
> >
> > ------------------------------------------------------------------------------------------------------------------------
> >
> > --- Event Stage 0: Main Stage
> >
> > MatMult 3927 1.0 2.4071e+01 1.3 6.14e+06 1.4 2.4e+04 1.3e+03
> > 0.0e+00 12 18 93 12 0 12 18 93 12 0 19
> > MatSolve 3967 1.0 2.5914e+00 1.9 7.99e+07 1.9 0.0e+00 0.0e+00
> > 0.0e+00 1 17 0 0 0 1 17 0 0 0 168
> > MatLUFactorNum 40 1.0 4.4779e-01 1.5 3.14e+07 1.5 0.0e+00 0.0e+00
> > 0.0e+00 0 2 0 0 0 0 2 0 0 0 85
> > MatILUFactorSym 2 1.0 3.1099e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> > 4.0e+00 0 0 0 0 0 0 0 0 0 0 0
> > MatScale 20 1.0 1.1487e-01 8.7 8.73e+07 8.9 0.0e+00 0.0e+00
> > 0.0e+00 0 0 0 0 0 0 0 0 0 0 39
> > MatAssemblyBegin 40 1.0 7.8844e+00 1.3 0.00e+00 0.0 7.6e+02 2.8e+05
> > 8.0e+01 4 0 3 83 1 4 0 3 83 1 0
> > MatAssemblyEnd 40 1.0 6.9408e+00 1.2 0.00e+00 0.0 1.2e+01 9.6e+02
> > 6.4e+01 4 0 0 0 1 4 0 0 0 1 0
> > MatGetOrdering 2 1.0 8.0509e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> > 6.0e+00 0 0 0 0 0 0 0 0 0 0 0
> > MatZeroEntries 21 1.0 1.4379e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
> > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> > VecMDot 3792 1.0 4.7372e+01 1.4 5.20e+06 1.4 0.0e+00 0.0e+00
> > 3.8e+03 24 29 0 0 30 24 29 0 0 30 15
> > VecNorm 3967 1.0 3.9513e+01 1.2 4.11e+05 1.2 0.0e+00 0.0e+00
> > 4.0e+03 21 2 0 0 32 21 2 0 0 32 1
> > VecScale 3947 1.0 3.4941e-02 1.2 2.18e+08 1.2 0.0e+00 0.0e+00
> > 0.0e+00 0 1 0 0 0 0 1 0 0 0 738
> > VecCopy 155 1.0 1.0029e-0125.1 0.00e+00 0.0 0.0e+00 0.0e+00
> > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> > VecSet 4142 1.0 3.4638e-01 6.6 0.00e+00 0.0 0.0e+00 0.0e+00
> > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> > VecAXPY 290 1.0 5.9618e-03 1.2 2.14e+08 1.2 0.0e+00 0.0e+00
> > 0.0e+00 0 0 0 0 0 0 0 0 0 0 709
> > VecMAXPY 3947 1.0 1.5566e+00 1.3 1.64e+08 1.3 0.0e+00 0.0e+00
> > 0.0e+00 1 31 0 0 0 1 31 0 0 0 498
> > VecAssemblyBegin 80 1.0 4.1793e+00 1.1 0.00e+00 0.0 9.6e+02 1.4e+04
> > 2.4e+02 2 0 4 5 2 2 0 4 5 2 0
> > VecAssemblyEnd 80 1.0 2.0682e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00
> > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> > VecScatterBegin 3927 1.0 2.8672e-01 3.9 0.00e+00 0.0 2.4e+04 1.3e+03
> > 0.0e+00 0 0 93 12 0 0 0 93 12 0 0
> > VecScatterEnd 3927 1.0 2.2135e+01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00
> > 0.0e+00 11 0 0 0 0 11 0 0 0 0 0
> > VecNormalize 3947 1.0 3.9593e+01 1.2 6.11e+05 1.2 0.0e+00 0.0e+00
> > 3.9e+03 21 3 0 0 32 21 3 0 0 32 2
> > KSPGMRESOrthog 3792 1.0 4.8670e+01 1.3 9.92e+06 1.3 0.0e+00 0.0e+00
> > 3.8e+03 25 58 0 0 30 25 58 0 0 30 30
> > KSPSetup 80 1.0 2.0014e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> > 2.0e+01 0 0 0 0 0 0 0 0 0 0 0
> > KSPSolve 40 1.0 1.0660e+02 1.0 5.90e+06 1.0 2.4e+04 1.3e+03
> > 1.2e+04 62100 93 12 97 62100 93 12 97 23
> > PCSetUp 80 1.0 4.5669e-01 1.5 3.05e+07 1.5 0.0e+00 0.0e+00
> > 1.4e+01 0 2 0 0 0 0 2 0 0 0 83
> > PCSetUpOnBlocks 40 1.0 4.5418e-01 1.5 3.07e+07 1.5 0.0e+00 0.0e+00
> > 1.0e+01 0 2 0 0 0 0 2 0 0 0 84
> > PCApply 3967 1.0 4.1737e+00 2.0 5.30e+07 2.0 0.0e+00 0.0e+00
> > 4.0e+03 2 17 0 0 32 2 17 0 0 32 104
> >
> > ------------------------------------------------------------------------------------------------------------------------
> >
> >
> >
> > Memory usage is given in bytes:
> >
> > Object Type Creations Destructions Memory Descendants' Mem.
> >
> > --- Event Stage 0: Main Stage
> >
> > Matrix 8 8 21136 0
> > Index Set 12 12 74952 0
> > Vec 81 81 1447476 0
> > Vec Scatter 2 2 0 0
> > Krylov Solver 4 4 33760 0
> > Preconditioner 4 4 392 0
> > ========================================================================================================================
> >
> > Average time to get PetscTime(): 1.09673e-06
> > Average time for MPI_Barrier(): 3.90053e-05
> > Average time for zero size MPI_Send(): 1.65105e-05
> > OptionTable: -log_summary
> > Compiled without FORTRAN kernels
> > Compiled with full precision matrices (default)
> > sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 4
> > sizeof(PetscScalar) 8
> > Configure run at: Thu Jan 18 12:23:31 2007
> > Configure options: --with-vendor-compilers=intel --with-x=0 --with-shared
> > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32
> > --with-mpi-dir=/opt/mpich/myrinet/intel/
> > -----------------------------------------
> > Libraries compiled on Thu Jan 18 12:24:41 SGT 2007 on atlas1.nus.edu.sg
> > Machine characteristics: Linux atlas1.nus.edu.sg 2.4.21-20.ELsmp #1 SMP
> > Wed Sep 8 17:29:34 GMT 2004 i686 i686 i386 GNU/Linux
> > Using PETSc directory: /nas/lsftmp/g0306332/petsc-2.3.2-p8
> > Using PETSc arch: linux-mpif90
> > -----------------------------------------
> > Using C compiler: /opt/mpich/myrinet/intel/bin/mpicc -fPIC -g
> > Using Fortran compiler: /opt/mpich/myrinet/intel/bin/mpif90 -I. -fPIC -g
> > -w90 -w
> > -----------------------------------------
> > Using include paths: -I/nas/lsftmp/g0306332/petsc-
> > 2.3.2-p8-I/nas/lsftmp/g0306332/petsc-
> > 2.3.2-p8/bmake/linux-mpif90 -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include
> > -I/opt/mpich/myrinet/intel/include
> > ------------------------------------------
> > Using C linker: /opt/mpich/myrinet/intel/bin/mpicc -fPIC -g
> > Using Fortran linker: /opt/mpich/myrinet/intel/bin/mpif90 -I. -fPIC -g
> > -w90 -w
> > Using libraries:
> > -Wl,-rpath,/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90
> > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 -lpetscts
> > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc
> > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32
> > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack -lmkl_ia32 -lguide
> > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib
> > -Wl,-rpath,/opt/mpich/myrinet/intel/lib -L/opt/mpich/myrinet/intel/lib
> > -Wl,-rpath,-rpath -Wl,-rpath,-ldl -L-ldl -lmpich -Wl,-rpath,-L -lgm
> > -lpthread -Wl,-rpath,/opt/intel/compiler70/ia32/lib
> > -Wl,-rpath,/opt/intel/compiler70/ia32/lib -L/opt/intel/compiler70/ia32/lib
> > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts -lcxa
> > -lunwind -ldl -lmpichf90 -Wl,-rpath,/opt/gm/lib -L/opt/gm/lib -lPEPCF90
> > -Wl,-rpath,/opt/intel/compiler70/ia32/lib -L/opt/intel/compiler70/ia32/lib
> > -Wl,-rpath,/usr/lib -L/usr/lib -lintrins -lIEPCF90 -lF90 -lm -Wl,-rpath,\
> > -Wl,-rpath,\ -L\ -ldl -lmpich -Wl,-rpath,\ -L\ -lgm -lpthread
> > -Wl,-rpath,/opt/intel/compiler70/ia32/lib -L/opt/intel/compiler70/ia32/lib
> > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts -lcxa -lunwind -ldl
> >
> >
> > This is the result I get for running 20 steps. There are 2 matrix to be
> > solved. I've only parallize the solving of linear equations and kept the
> > rest of the code serial for this test. However, I found that it's much
> > slower than the sequential version.
> >
> > From the ratio, it seems that MatScale and VecSet 's ratio are very high.
> > I've done a scaling of 0.5 for momentum eqn. Is that the reason for the
> > slowness? That is all I can decipher ....
> >
> > Thank you.
> >
> >
> >
> >
> >
> > On 2/10/07, Matthew Knepley <knepley at gmail.com> wrote:
> > >
> > > On 2/9/07, Ben Tay < zonexo at gmail.com> wrote:
> > > >
> > > > ops.... it worked for ex2 and ex2f ;-)
> > > >
> > > > so what could be wrong? is there some commands or subroutine which i
> > > > must call? btw, i'm programming in fortran.
> > > >
> > >
> > > Yes, you must call PetscFinalize() in your code.
> > >
> > > Matt
> > >
> > >
> > > thank you.
> > > >
> > > >
> > > > On 2/9/07, Matthew Knepley <knepley at gmail.com > wrote:
> > > > >
> > > > > Problems do not go away by ignoring them. Something is wrong here,
> > > > > and it may
> > > > > affect the rest of your program. Please try to run an example:
> > > > >
> > > > > cd src/ksp/ksp/examples/tutorials
> > > > > make ex2
> > > > > ./ex2 -log_summary
> > > > >
> > > > > Matt
> > > > >
> > > > > On 2/9/07, Ben Tay <zonexo at gmail.com> wrote:
> > > > > >
> > > > > > Well, I don't know what's wrong. I did the same thing for -info
> > > > > > and it worked. Anyway, is there any other way?
> > > > > >
> > > > > > Like I can use -mat_view or call matview( ... ) to view a matrix.
> > > > > > Is there a similar subroutine for me to call?
> > > > > >
> > > > > > Thank you.
> > > > > >
> > > > > >
> > > > > > On 2/9/07, Matthew Knepley <knepley at gmail.com > wrote:
> > > > > > >
> > > > > > > Impossible, please check the spelling, and make sure your
> > > > > > > command line was not truncated.
> > > > > > >
> > > > > > > Matt
> > > > > > >
> > > > > > > On 2/9/07, Ben Tay < zonexo at gmail.com> wrote:
> > > > > > > >
> > > > > > > > ya, i did use -log_summary. but no output.....
> > > > > > > >
> > > > > > > > On 2/9/07, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -log_summary
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, 9 Feb 2007, Ben Tay wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I've tried to use log_summary but nothing came out? Did I
> > > > > > > > > miss out
> > > > > > > > > > something? It worked when I used -info...
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 2/9/07, Lisandro Dalcin <dalcinl at gmail.com > wrote:
> > > > > > > > > > >
> > > > > > > > > > > On 2/8/07, Ben Tay < zonexo at gmail.com> wrote:
> > > > > > > > > > > > i'm trying to solve my cfd code using PETSc in
> > > > > > > > > parallel. Besides the
> > > > > > > > > > > linear
> > > > > > > > > > > > eqns for PETSc, other parts of the code has also been
> > > > > > > > > parallelized using
> > > > > > > > > > > > MPI.
> > > > > > > > > > >
> > > > > > > > > > > Finite elements or finite differences, or what?
> > > > > > > > > > >
> > > > > > > > > > > > however i find that the parallel version of the code
> > > > > > > > > running on 4
> > > > > > > > > > > processors
> > > > > > > > > > > > is even slower than the sequential version.
> > > > > > > > > > >
> > > > > > > > > > > Can you monitor the convergence and iteration count of
> > > > > > > > > momentum and
> > > > > > > > > > > poisson steps?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > in order to find out why, i've used the -info option
> > > > > > > > > to print out the
> > > > > > > > > > > > details. there are 2 linear equations being solved -
> > > > > > > > > momentum and
> > > > > > > > > > > poisson.
> > > > > > > > > > > > the momentum one is twice the size of the poisson. it
> > > > > > > > > is shown below:
> > > > > > > > > > >
> > > > > > > > > > > Can you use -log_summary command line option and send
> > > > > > > > > the output attached?
> > > > > > > > > > >
> > > > > > > > > > > > i saw some statements stating "seq". am i running in
> > > > > > > > > sequential or
> > > > > > > > > > > parallel
> > > > > > > > > > > > mode? have i preallocated too much space?
> > > > > > > > > > >
> > > > > > > > > > > It seems you are running in parallel. The "Seq" are
> > > > > > > > > related to local,
> > > > > > > > > > > internal objects. In PETSc, parallel matrices have inner
> > > > > > > > > sequential
> > > > > > > > > > > matrices.
> > > > > > > > > > >
> > > > > > > > > > > > lastly, if Ax=b, A_sta and A_end
> > > > > > > > > from MatGetOwnershipRange and b_sta
> > > > > > > > > > > and
> > > > > > > > > > > > b_end from VecGetOwnershipRange should always be the
> > > > > > > > > same value, right?
> > > > > > > > > > >
> > > > > > > > > > > I should. If not, you are likely going to get an runtime
> > > > > > > > > error.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Lisandro Dalcín
> > > > > > > > > > > ---------------
> > > > > > > > > > > Centro Internacional de Métodos Computacionales en
> > > > > > > > > Ingeniería (CIMEC)
> > > > > > > > > > > Instituto de Desarrollo Tecnológico para la Industria
> > > > > > > > > Química (INTEC)
> > > > > > > > > > > Consejo Nacional de Investigaciones Científicas y
> > > > > > > > > Técnicas (CONICET)
> > > > > > > > > > > PTLC - Güemes 3450, (3000) Santa Fe, Argentina
> > > > > > > > > > > Tel/Fax: +54-(0)342-451.1594
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > One trouble is that despite this system, anyone who reads
> > > > > > > journals widely
> > > > > > > and critically is forced to realize that there are scarcely any
> > > > > > > bars to eventual
> > > > > > > publication. There seems to be no study too fragmented, no
> > > > > > > hypothesis too
> > > > > > > trivial, no literature citation too biased or too egotistical,
> > > > > > > no design too
> > > > > > > warped, no methodology too bungled, no presentation of results
> > > > > > > too
> > > > > > > inaccurate, too obscure, and too contradictory, no analysis too
> > > > > > > self-serving,
> > > > > > > no argument too circular, no conclusions too trifling or too
> > > > > > > unjustified, and
> > > > > > > no grammar and syntax too offensive for a paper to end up in
> > > > > > > print. -- Drummond Rennie
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > One trouble is that despite this system, anyone who reads journals
> > > > > widely
> > > > > and critically is forced to realize that there are scarcely any bars
> > > > > to eventual
> > > > > publication. There seems to be no study too fragmented, no
> > > > > hypothesis too
> > > > > trivial, no literature citation too biased or too egotistical, no
> > > > > design too
> > > > > warped, no methodology too bungled, no presentation of results too
> > > > > inaccurate, too obscure, and too contradictory, no analysis too
> > > > > self-serving,
> > > > > no argument too circular, no conclusions too trifling or too
> > > > > unjustified, and
> > > > > no grammar and syntax too offensive for a paper to end up in print.
> > > > > -- Drummond Rennie
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > One trouble is that despite this system, anyone who reads journals
> > > widely
> > > and critically is forced to realize that there are scarcely any bars to
> > > eventual
> > > publication. There seems to be no study too fragmented, no hypothesis
> > > too
> > > trivial, no literature citation too biased or too egotistical, no design
> > > too
> > > warped, no methodology too bungled, no presentation of results too
> > > inaccurate, too obscure, and too contradictory, no analysis too
> > > self-serving,
> > > no argument too circular, no conclusions too trifling or too
> > > unjustified, and
> > > no grammar and syntax too offensive for a paper to end up in print. --
> > > Drummond Rennie
> > >
> >
> >
>
>
>
More information about the petsc-users
mailing list