PETSc CG solver uses more iterations than other CG solver
Berend van Wachem
berend at chalmers.se
Sat Mar 10 14:39:28 CST 2007
Hi,
I have also used the VOF/level set model in our PETSc based code, and it does
require a lot more iterations than a single phase model because of the
stencil. I guess you are solving
d/dx 1/rho d/dx p = 0
? Which can result in a terrible coefficient structure.
Are you sure the solver/preconditioner you use with PETSc are the same as in
your other code?
I think for the problem size, you are having an awful lot of iterations! Are
you sure the coefficients are put in the right location in the matrix?
Berend.
>
> One boundary cell is defined to have constant pressure, since that
> makes the equation system have a unique solution. I tried your
> command, and it lowered the number of iterations for most of the time
> steps, but for some it reached the maximum number of iterations
> (10000) without converging.
>
> I also tried making all the boundaries von Neumann and using your
> command. That made the number of iterations more constant, instead of
> varying between 700 and 2000, it stayed on around 1200. But it
> actually increased the average number of iterations somewhat. Still
> far from the performance of the other solver.
> I've also checked the convergence criteria, and it is the same for
> both solvers.
>
> Siterer Lisandro Dalcin <dalcinl at gmail.com>:
> > On 3/9/07, Knut Erik Teigen <knutert at stud.ntnu.no> wrote:
> >> To solve the Navier-Stokes equations, I use an explicit Runge-Kutta
> >> method with Chorin's projection method,
> >> so a Poisson equation with von Neumann boundary conditions for the
> >> pressure has to be solved at every time-step.
> >
> > All boundary conditions are Neumann type? In that case, please try to
> > run your program with the following command line option:
> >
> > -ksp_constant_null_space
> >
> > and let me know if this corrected your problem.
> >
> >
> > The equation system is
> >
> >> positive definite, so I use the CG solver with the ICC preconditioner.
> >> The problem is that the PETSc solver seems to need a lot more iterations
> >> to reach the solution than another CG solver I'm using. On a small test
> >> problem(a rising bubble) with a 60x40 grid, the PETSc solver needs over
> >> 1000 iterations on average, while the other solver needs less than 100.
> >> I am using KSPSetInitialGuessNonzero, without this the number of
> >> iterations is even higher.
> >> I have also tried applying PETSc to a similar problem, solving the
> >> Poisson equation with von Neumann boundaries and a forcing function of
> >> f=sin(pi * x)+sin(pi *y). For this problem, the number of iterations is
> >> almost exactly the same for PETSc and the other solver.
> >>
> >> Does anyone know what the problem might be? Any help is greatly
> >> appreciated. I've included the -ksp_view of one of the time steps
> >> and the -log_summary below.
> >>
> >> Regards,
> >> Knut Erik Teigen
> >> MSc student
> >> Norwegian University of Science and Technology
> >>
> >> Output from -ksp_view:
> >>
> >> KSP Object:
> >> type: cg
> >> maximum iterations=10000
> >> tolerances: relative=1e-06, absolute=1e-50, divergence=10000
> >> left preconditioning
> >> PC Object:
> >> type: icc
> >> ICC: 0 levels of fill
> >> ICC: factor fill ratio allocated 1
> >> ICC: factor fill ratio needed 0.601695
> >> Factored matrix follows
> >> Matrix Object:
> >> type=seqsbaij, rows=2400, cols=2400
> >> total: nonzeros=7100, allocated nonzeros=7100
> >> block size is 1
> >> linear system matrix = precond matrix:
> >> Matrix Object:
> >> type=seqaij, rows=2400, cols=2400
> >> total: nonzeros=11800, allocated nonzeros=12000
> >> not using I-node routines
> >> Poisson converged after 1403 iterations
> >>
> >> Output from -log_summary
> >>
> >> ---------------------------------------------- PETSc Performance
> >> Summary: ----------------------------------------------
> >>
> >> ./run on a gcc-ifc-d named iept0415 with 1 processor, by knutert Fri Mar
> >> 9 17:06:05 2007
> >> Using Petsc Release Version 2.3.2, Patch 8, Tue Jan 2 14:33:59 PST 2007
> >> HG revision: ebeddcedcc065e32fc252af32cf1d01ed4fc7a80
> >>
> >> Max Max/Min Avg Total
> >> Time (sec): 5.425e+02 1.00000 5.425e+02
> >> Objects: 7.000e+02 1.00000 7.000e+02
> >> Flops: 6.744e+10 1.00000 6.744e+10 6.744e+10
> >> Flops/sec: 1.243e+08 1.00000 1.243e+08 1.243e+08
> >> Memory: 4.881e+05 1.00000 4.881e+05
> >> MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00
> >> MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00
> >> MPI Reductions: 1.390e+03 1.00000
> >>
> >> Flop counting convention: 1 flop = 1 real number operation of type
> >> (multiply/divide/add/subtract)
> >> e.g., VecAXPY() for real vectors of length N
> >> --> 2N flops
> >> and VecAXPY() for complex vectors of length
> >> N --> 8N flops
> >>
> >> Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages
> >> --- -- Message Lengths -- -- Reductions --
> >> Avg %Total Avg %Total counts %
> >> Total Avg %Total counts %Total
> >> 0: Main Stage: 5.4246e+02 100.0% 6.7437e+10 100.0% 0.000e+00
> >> 0.0% 0.000e+00 0.0% 1.390e+03 100.0%
> >>
> >> ------------------------------------------------------------------------
> >>------------------------------------------------ See the 'Profiling'
> >> chapter of the users' manual for details on interpreting output.
> >> Phase summary info:
> >> Count: number of times phase was executed
> >> Time and Flops/sec: Max - maximum over all processors
> >> Ratio - ratio of maximum to minimum over all
> >> processors
> >> Mess: number of messages sent
> >> Avg. len: average message length
> >> Reduct: number of global reductions
> >> Global: entire computation
> >> Stage: stages of a computation. Set stages with PetscLogStagePush()
> >> and PetscLogStagePop().
> >> %T - percent time in this phase %F - percent flops in this
> >> phase
> >> %M - percent messages in this phase %L - percent message
> >> lengths in this phase
> >> %R - percent reductions in this phase
> >> Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time
> >> over all processors)
> >> ------------------------------------------------------------------------
> >>------------------------------------------------
> >>
> >>
> >> ##########################################################
> >> # #
> >> # WARNING!!! #
> >> # #
> >> # This code was compiled with a debugging option, #
> >> # To get timing results run config/configure.py #
> >> # using --with-debugging=no, the performance will #
> >> # be generally two or three times faster. #
> >> # #
> >> ##########################################################
> >>
> >>
> >>
> >>
> >> ##########################################################
> >> # #
> >> # WARNING!!! #
> >> # #
> >> # This code was run without the PreLoadBegin() #
> >> # macros. To get timing results we always recommend #
> >> # preloading. otherwise timing numbers may be #
> >> # meaningless. #
> >> ##########################################################
> >>
> >>
> >> Event Count Time (sec) Flops/sec
> >> --- Global --- --- Stage --- Total
> >> Max Ratio Max Ratio Max Ratio Mess Avg len
> >> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
> >> ------------------------------------------------------------------------
> >>------------------------------------------------
> >>
> >> --- Event Stage 0: Main Stage
> >>
> >> VecDot 1668318 1.0 1.9186e+01 1.0 4.17e+08 1.0 0.0e+00 0.0e+00
> >> 0.0e+00 4 12 0 0 0 4 12 0 0 0 417
> >> VecNorm 835191 1.0 1.0935e+01 1.0 3.67e+08 1.0 0.0e+00 0.0e+00
> >> 0.0e+00 2 6 0 0 0 2 6 0 0 0 367
> >> VecCopy 688 1.0 1.6126e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> >> VecAXPY 1667630 1.0 2.4141e+01 1.0 3.32e+08 1.0 0.0e+00 0.0e+00
> >> 0.0e+00 4 12 0 0 0 4 12 0 0 0 332
> >> VecAYPX 833815 1.0 1.9062e+01 1.0 2.10e+08 1.0 0.0e+00 0.0e+00
> >> 0.0e+00 4 6 0 0 0 4 6 0 0 0 210
> >> VecAssemblyBegin 688 1.0 8.5354e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> >> VecAssemblyEnd 688 1.0 7.8177e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> >> MatMult 834503 1.0 1.5044e+02 1.0 1.18e+08 1.0 0.0e+00 0.0e+00
> >> 0.0e+00 28 26 0 0 0 28 26 0 0 0 118
> >> MatSolve 835191 1.0 1.8130e+02 1.0 1.42e+08 1.0 0.0e+00 0.0e+00
> >> 0.0e+00 33 38 0 0 0 33 38 0 0 0 142
> >> MatCholFctrNum 1 1.0 1.1630e-03 1.0 2.06e+06 1.0 0.0e+00 0.0e+00
> >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 2
> >> MatICCFactorSym 1 1.0 2.9588e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> >> 2.0e+00 0 0 0 0 0 0 0 0 0 0 0
> >> MatAssemblyBegin 688 1.0 2.6343e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> >> MatAssemblyEnd 688 1.0 1.0964e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> >> MatGetOrdering 1 1.0 2.6798e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> >> 2.0e+00 0 0 0 0 0 0 0 0 0 0 0
> >> KSPSetup 1 1.0 2.7585e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> >> 6.0e+00 0 0 0 0 0 0 0 0 0 0 0
> >> KSPSolve 688 1.0 4.2809e+02 1.0 1.58e+08 1.0 0.0e+00 0.0e+00
> >> 1.4e+03 79100 0 0100 79100 0 0100 158
> >> PCSetUp 1 1.0 1.7900e-03 1.0 1.34e+06 1.0 0.0e+00 0.0e+00
> >> 4.0e+00 0 0 0 0 0 0 0 0 0 0 1
> >> PCApply 835191 1.0 1.8864e+02 1.0 1.36e+08 1.0 0.0e+00 0.0e+00
> >> 0.0e+00 35 38 0 0 0 35 38 0 0 0 136
> >> ------------------------------------------------------------------------
> >>------------------------------------------------
> >>
> >> Memory usage is given in bytes:
> >>
> >> Object Type Creations Destructions Memory Descendants'
> >> Mem.
> >>
> >> --- Event Stage 0: Main Stage
> >>
> >> Index Set 2 2 19640 0
> >> Vec 694 693 299376 0
> >> Matrix 2 2 56400 0
> >> Krylov Solver 1 1 36 0
> >> Preconditioner 1 1 108 0
> >> ========================================================================
> >>================================================ Average time to get
> >> PetscTime(): 2.86102e-07
> >> OptionTable: -ksp_type cg
> >> OptionTable: -log_summary -ksp_view
> >> OptionTable: -pc_type icc
> >> Compiled without FORTRAN kernels
> >> Compiled with full precision matrices (default)
> >> sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 4
> >> sizeof(PetscScalar) 8
> >> Configure run at: Thu Mar 8 11:54:22 2007
> >> Configure options: --with-cc=gcc --with-fc=ifort
> >> --download-f-blas-lapack=1 --download-mpich=1 --with-debugging=1
> >> --with-shared=0
> >> -----------------------------------------
> >> Libraries compiled on Thu Mar 8 12:08:22 CET 2007 on iept0415
> >> Machine characteristics: Linux iept0415 2.6.16.21-0.8-default #1 Mon Jul
> >> 3 18:25:39 UTC 2006 i686 i686 i386 GNU/Linux
> >> Using PETSc directory: /opt/petsc-2.3.2-p8
> >> Using PETSc arch: gcc-ifc-debug
> >> -----------------------------------------
> >> Using C compiler: gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
> >> -g3
> >> Using Fortran compiler: ifort -fPIC -g
> >> -----------------------------------------
> >> Using include paths: -I/opt/petsc-2.3.2-p8
> >> -I/opt/petsc-2.3.2-p8/bmake/gcc-ifc-debug -I/opt/petsc-2.3.2-p8/include
> >> -I/opt/petsc-2.3.2-p8/externalpackages/mpich2-1.0.4p1/gcc-ifc-debug/incl
> >>ude -I/usr/X11R6/include
> >> ------------------------------------------
> >> Using C linker: gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -g3
> >> Using Fortran linker: ifort -fPIC -g
> >> Using libraries: -Wl,-rpath,/opt/petsc-2.3.2-p8/lib/gcc-ifc-debug
> >> -L/opt/petsc-2.3.2-p8/lib/gcc-ifc-debug -lpetscts -lpetscsnes -lpetscksp
> >> -lpetscdm -lpetscmat -lpetscvec -lpetsc
> >> -Wl,-rpath,/opt/petsc-2.3.2-p8/externalpackages/mpich2-1.0.4p1/gcc-ifc-d
> >>ebug/lib
> >> -L/opt/petsc-2.3.2-p8/externalpackages/mpich2-1.0.4p1/gcc-ifc-debug/lib
> >> -lmpich -lnsl -lrt -L/usr/X11R6/lib -lX11
> >> -Wl,-rpath,/opt/petsc-2.3.2-p8/externalpackages/fblaslapack/gcc-ifc-debu
> >>g -L/opt/petsc-2.3.2-p8/externalpackages/fblaslapack/gcc-ifc-debug
> >> -lflapack
> >> -Wl,-rpath,/opt/petsc-2.3.2-p8/externalpackages/fblaslapack/gcc-ifc-debu
> >>g -L/opt/petsc-2.3.2-p8/externalpackages/fblaslapack/gcc-ifc-debug
> >> -lfblas -lm -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0
> >> -L/usr/lib/gcc/i586-suse-linux/4.1.0
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0/../../../../i586-suse-linu
> >>x/lib
> >> -L/usr/lib/gcc/i586-suse-linux/4.1.0/../../../../i586-suse-linux/lib
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0/../../..
> >> -L/usr/lib/gcc/i586-suse-linux/4.1.0/../../.. -ldl -lgcc_s
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0/../../../../i586-suse-linu
> >>x/lib -Wl,-rpath,/usr/lib/gcc/!
> >
> > i5!
> >
> >> 86-suse-linux/4.1.0/../../.. -Wl,-rpath,/opt/intel/fc/9.1.036/lib
> >> -L/opt/intel/fc/9.1.036/lib
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0/
> >> -L/usr/lib/gcc/i586-suse-linux/4.1.0/
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0/../../../
> >> -L/usr/lib/gcc/i586-suse-linux/4.1.0/../../../ -lifport -lifcore
> >> -limf -lm -lipgo -lirc -lirc_s
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0/../../../../i586-suse-linu
> >>x/lib -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0/../../.. -lm
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0
> >> -L/usr/lib/gcc/i586-suse-linux/4.1.0
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0/../../../../i586-suse-linu
> >>x/lib
> >> -L/usr/lib/gcc/i586-suse-linux/4.1.0/../../../../i586-suse-linux/lib
> >> -Wl,-rpath,/usr/lib/gcc/i586-suse-linux/4.1.0/../../..
> >> -L/usr/lib/gcc/i586-suse-linux/4.1.0/../../.. -ldl -lgcc_s -ldl
> >
> > --
> > Lisandro Dalcín
> > ---------------
> > Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
> > Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
> > Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
> > PTLC - Güemes 3450, (3000) Santa Fe, Argentina
> > Tel/Fax: +54-(0)342-451.1594
More information about the petsc-users
mailing list