Smoother settings for AMG
Matthew Knepley
knepley at gmail.com
Wed Jul 29 16:00:08 CDT 2009
On Wed, Jul 29, 2009 at 3:54 PM, BAYRAKTAR Harun <Harun.BAYRAKTAR at 3ds.com>wrote:
> Hi,
>
> I am trying to solve a system of equations and I am having difficulty
> picking the right smoothers for AMG (using ML as pc_type) in PETSc for
> parallel execution. First here is what happens in terms of CG (ksp_type)
> iteration counts (both columns use block jacobi):
Are you sure you have an elliptic system? These iteration counts are
extremely
high.
Matt
>
> cpus | AMG w/ ICC(0) x1 | AMG w/ SOR x4
> ------------------------------------------------------
> 1 | 43 | 243
> 4 | 699 | 379
>
> x1 or x4 means 1 or 4 iterations of smoother application at each AMG
> level (all details from ksp view for the 4 cpu run are below). The main
> observation is that on 1 cpu, AMG w/ ICC(0) is a clear winner but falls
> apart in parallel. SOR on the other hand experiences a 1.5X increase in
> iteration count which is totally expected from the quality of coarsening
> ML delivers in parallel.
>
> I basically would like to find a way (if possible) to have the number of
> iterations in parallel stay with 1-2X of 1 cpu iteration count for the
> AMG w/ ICC case. Is there a way to achieve this?
>
> Thanks,
> Harun
>
> %%%%%%%%%%%%%%%%%%%%%%%%%
> AMG w/ ICC(0) x1 ksp_view
> %%%%%%%%%%%%%%%%%%%%%%%%%
> KSP Object:
> type: cg
> maximum iterations=10000
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:
> type: ml
> MG: type is MULTIPLICATIVE, levels=3 cycles=v, pre-smooths=1,
> post-smooths=1
> Coarse gride solver -- level 0 -------------------------------
> KSP Object:(mg_coarse_)
> type: preonly
> maximum iterations=1, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_coarse_)
> type: redundant
> Redundant preconditioner: First (color=0) of 4 PCs follows
> KSP Object:(mg_coarse_redundant_)
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_coarse_redundant_)
> type: lu
> LU: out-of-place factorization
> matrix ordering: nd
> LU: tolerance for zero pivot 1e-12
> LU: factor fill ratio needed 2.17227
> Factored matrix follows
> Matrix Object:
> type=seqaij, rows=283, cols=283
> total: nonzeros=21651, allocated nonzeros=21651
> using I-node routines: found 186 nodes, limit used is
> 5
> linear system matrix = precond matrix:
> Matrix Object:
> type=seqaij, rows=283, cols=283
> total: nonzeros=9967, allocated nonzeros=14150
> not using I-node routines
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=283, cols=283
> total: nonzeros=9967, allocated nonzeros=9967
> not using I-node (on process 0) routines
> Down solver (pre-smoother) on level 1 -------------------------------
> KSP Object:(mg_levels_1_)
> type: richardson
> Richardson: damping factor=0.9
> maximum iterations=1, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_1_)
> type: bjacobi
> block Jacobi: number of blocks = 4
> Local solve is same for all blocks, in the following KSP and PC
> objects:
> KSP Object:(mg_levels_1_sub_)
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_1_sub_)
> type: icc
> ICC: 0 levels of fill
> ICC: factor fill ratio allocated 1
> ICC: using Manteuffel shift
> ICC: factor fill ratio needed 0.514899
> Factored matrix follows
> Matrix Object:
> type=seqsbaij, rows=2813, cols=2813
> total: nonzeros=48609, allocated nonzeros=48609
> block size is 1
> linear system matrix = precond matrix:
> Matrix Object:
> type=seqaij, rows=2813, cols=2813
> total: nonzeros=94405, allocated nonzeros=94405
> not using I-node routines
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=10654, cols=10654
> total: nonzeros=376634, allocated nonzeros=376634
> not using I-node (on process 0) routines
> Up solver (post-smoother) on level 1 -------------------------------
> KSP Object:(mg_levels_1_)
> type: richardson
> Richardson: damping factor=0.9
> maximum iterations=1
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_1_)
> type: bjacobi
> block Jacobi: number of blocks = 4
> Local solve is same for all blocks, in the following KSP and PC
> objects:
> KSP Object:(mg_levels_1_sub_)
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_1_sub_)
> type: icc
> ICC: 0 levels of fill
> ICC: factor fill ratio allocated 1
> ICC: using Manteuffel shift
> ICC: factor fill ratio needed 0.514899
> Factored matrix follows
> Matrix Object:
> type=seqsbaij, rows=2813, cols=2813
> total: nonzeros=48609, allocated nonzeros=48609
> block size is 1
> linear system matrix = precond matrix:
> Matrix Object:
> type=seqaij, rows=2813, cols=2813
> total: nonzeros=94405, allocated nonzeros=94405
> not using I-node routines
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=10654, cols=10654
> total: nonzeros=376634, allocated nonzeros=376634
> not using I-node (on process 0) routines
> Down solver (pre-smoother) on level 2 -------------------------------
> KSP Object:(mg_levels_2_)
> type: richardson
> Richardson: damping factor=0.9
> maximum iterations=1, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_2_)
> type: bjacobi
> block Jacobi: number of blocks = 4
> Local solve is same for all blocks, in the following KSP and PC
> objects:
> KSP Object:(mg_levels_2_sub_)
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_2_sub_)
> type: icc
> ICC: 0 levels of fill
> ICC: factor fill ratio allocated 1
> ICC: using Manteuffel shift
> ICC: factor fill ratio needed 0.519045
> Factored matrix follows
> Matrix Object:
> type=seqsbaij, rows=101164, cols=101164
> total: nonzeros=1378558, allocated nonzeros=1378558
> block size is 1
> linear system matrix = precond matrix:
> Matrix Object:
> type=seqaij, rows=101164, cols=101164
> total: nonzeros=2655952, allocated nonzeros=5159364
> not using I-node routines
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=411866, cols=411866
> total: nonzeros=10941434, allocated nonzeros=42010332
> not using I-node (on process 0) routines
> Up solver (post-smoother) on level 2 -------------------------------
> KSP Object:(mg_levels_2_)
> type: richardson
> Richardson: damping factor=0.9
> maximum iterations=1
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_2_)
> type: bjacobi
> block Jacobi: number of blocks = 4
> Local solve is same for all blocks, in the following KSP and PC
> objects:
> KSP Object:(mg_levels_2_sub_)
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_2_sub_)
> type: icc
> ICC: 0 levels of fill
> ICC: factor fill ratio allocated 1
> ICC: using Manteuffel shift
> ICC: factor fill ratio needed 0.519045
> Factored matrix follows
> Matrix Object:
> type=seqsbaij, rows=101164, cols=101164
> total: nonzeros=1378558, allocated nonzeros=1378558
> block size is 1
> linear system matrix = precond matrix:
> Matrix Object:
> type=seqaij, rows=101164, cols=101164
> total: nonzeros=2655952, allocated nonzeros=5159364
> not using I-node routines
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=411866, cols=411866
> total: nonzeros=10941434, allocated nonzeros=42010332
> not using I-node (on process 0) routines
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=411866, cols=411866
> total: nonzeros=10941434, allocated nonzeros=42010332
> not using I-node (on process 0) routines
>
> %%%%%%%%%%%%%%%%%%%%%%
> AMG w/ SOR x4 ksp_view
> %%%%%%%%%%%%%%%%%%%%%%
>
> KSP Object:
> type: cg
> maximum iterations=10000
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:
> type: ml
> MG: type is MULTIPLICATIVE, levels=3 cycles=v, pre-smooths=1,
> post-smooths=1
> Coarse gride solver -- level 0 -------------------------------
> KSP Object:(mg_coarse_)
> type: preonly
> maximum iterations=1, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_coarse_)
> type: redundant
> Redundant preconditioner: First (color=0) of 4 PCs follows
> KSP Object:(mg_coarse_redundant_)
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_coarse_redundant_)
> type: lu
> LU: out-of-place factorization
> matrix ordering: nd
> LU: tolerance for zero pivot 1e-12
> LU: factor fill ratio needed 2.17227
> Factored matrix follows
> Matrix Object:
> type=seqaij, rows=283, cols=283
> total: nonzeros=21651, allocated nonzeros=21651
> using I-node routines: found 186 nodes, limit used is
> 5
> linear system matrix = precond matrix:
> Matrix Object:
> type=seqaij, rows=283, cols=283
> total: nonzeros=9967, allocated nonzeros=14150
> not using I-node routines
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=283, cols=283
> total: nonzeros=9967, allocated nonzeros=9967
> not using I-node (on process 0) routines
> Down solver (pre-smoother) on level 1 -------------------------------
> KSP Object:(mg_levels_1_)
> type: richardson
> Richardson: damping factor=1
> maximum iterations=4, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_1_)
> type: sor
> SOR: type = local_symmetric, iterations = 1, omega = 1
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=10654, cols=10654
> total: nonzeros=376634, allocated nonzeros=376634
> not using I-node (on process 0) routines
> Up solver (post-smoother) on level 1 -------------------------------
> KSP Object:(mg_levels_1_)
> type: richardson
> Richardson: damping factor=1
> maximum iterations=4
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_1_)
> type: sor
> SOR: type = local_symmetric, iterations = 1, omega = 1
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=10654, cols=10654
> total: nonzeros=376634, allocated nonzeros=376634
> not using I-node (on process 0) routines
> Down solver (pre-smoother) on level 2 -------------------------------
> KSP Object:(mg_levels_2_)
> type: richardson
> Richardson: damping factor=1
> maximum iterations=4, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_2_)
> type: sor
> SOR: type = local_symmetric, iterations = 1, omega = 1
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=411866, cols=411866
> total: nonzeros=10941434, allocated nonzeros=42010332
> not using I-node (on process 0) routines
> Up solver (post-smoother) on level 2 -------------------------------
> KSP Object:(mg_levels_2_)
> type: richardson
> Richardson: damping factor=1
> maximum iterations=4
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> PC Object:(mg_levels_2_)
> type: sor
> SOR: type = local_symmetric, iterations = 1, omega = 1
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=411866, cols=411866
> total: nonzeros=10941434, allocated nonzeros=42010332
> not using I-node (on process 0) routines
> linear system matrix = precond matrix:
> Matrix Object:
> type=mpiaij, rows=411866, cols=411866
> total: nonzeros=10941434, allocated nonzeros=42010332
> not using I-node (on process 0) routines
>
>
>
--
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090729/fbf7e23e/attachment-0001.htm>
More information about the petsc-users
mailing list