[petsc-users] GAMG speed
Jed Brown
jedbrown at mcs.anl.gov
Tue Aug 13 22:36:56 CDT 2013
Michele Rosso <mrosso at uci.edu> writes:
> Hi Jed,
>
> I attached the output for both the runs you suggested. At the beginning
> of each file I included the options I used.
>
> On a side note, I tried to run with a grid of 256^3 (exactly as before)
> but with less levels, i.e. 3 instead of 4 or 5.
> My system stops the run because of an Out Of Memory condition. It is
> really odd since I have not changed anything except
> - pc_mg_levels. I cannot send you any output since there is none. Do
> you have any guess where the problem comes from?
The selected algorithm does a direct solve on the coarse grid. Each
time you reduce the number of levels, the coarse grid size grows by a
factor of 8. Going from 5 to 3 levels is going from a 16^3 coarse grid
to a 64^3 coarse grid. Applying a direct solver to the latter ends up
using a lot of memory. I think this is not worth bothering with and it
might even be (slightly) faster to use 6 levels. That is not where the
time is being spent.
> -log_summary -ksp_monitor -ksp_view -ksp_converged_reason -pc_type mg
> -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson
> -mg_levels_ksp_max_it 1
>
>
>
> 0 KSP Residual norm 3.653965664551e-05
> 1 KSP Residual norm 1.910638846094e-06
> 2 KSP Residual norm 8.690440116045e-08
> 3 KSP Residual norm 3.732213639394e-09
> 4 KSP Residual norm 1.964855338020e-10
This converges well.
> Max Max/Min Avg Total
> Time (sec): 4.048e+00 1.00012 4.048e+00
> Objects: 2.490e+02 1.00000 2.490e+02
> Flops: 2.663e+08 1.00000 2.663e+08 2.130e+09
> Flops/sec: 6.579e+07 1.00012 6.579e+07 5.263e+08
> MPI Messages: 6.820e+02 1.00000 6.820e+02 5.456e+03
> MPI Message Lengths: 8.245e+06 1.00000 1.209e+04 6.596e+07
> MPI Reductions: 4.580e+02 1.00000
> VecTDot 12 1.0 2.9428e-02 1.2 6.29e+06 1.0 0.0e+00 0.0e+00 1.2e+01 1 2 0 0 3 1 2 0 0 3 1710
> VecNorm 9 1.0 1.0796e-02 1.2 4.72e+06 1.0 0.0e+00 0.0e+00 9.0e+00 0 2 0 0 2 0 2 0 0 2 3497
> VecScale 24 1.0 2.4652e-04 1.1 1.99e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6442
> VecCopy 3 1.0 5.0740e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> VecSet 116 1.0 1.4349e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> VecAXPY 12 1.0 2.8027e-02 1.0 6.29e+06 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 1796
> VecAYPX 29 1.0 3.0655e-02 1.4 4.16e+06 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 1085
> VecScatterBegin 123 1.0 3.5391e-02 1.1 0.00e+00 0.0 3.5e+03 1.2e+04 0.0e+00 1 0 65 66 0 1 0 65 66 0 0
> VecScatterEnd 123 1.0 2.5395e-02 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> MatMult 31 1.0 2.3556e-01 1.0 5.62e+07 1.0 1.0e+03 2.3e+04 0.0e+00 6 21 19 36 0 6 21 19 36 0 1908
> MatMultAdd 24 1.0 5.9044e-02 1.0 1.21e+07 1.0 5.8e+02 2.8e+03 0.0e+00 1 5 11 2 0 1 5 11 2 0 1644
> MatMultTranspose 28 1.0 7.4601e-02 1.1 1.42e+07 1.0 6.7e+02 2.8e+03 0.0e+00 2 5 12 3 0 2 5 12 3 0 1518
> MatSolve 6 1.0 3.8311e-03 1.0 1.44e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 3006
> MatSOR 48 1.0 5.8050e-01 1.0 1.01e+08 1.0 8.6e+02 1.5e+04 4.8e+01 14 38 16 19 10 14 38 16 19 11 1390
Most of the solve time is in MatSOR and MatMult. That's expected since
the subdomains are pretty big.
> MatLUFactorSym 1 1.0 3.0620e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 1 0 0 0 0 1 0
> MatLUFactorNum 1 1.0 2.4665e-02 1.0 1.95e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 7 0 0 0 1 7 0 0 0 6329
> MatAssemblyBegin 20 1.0 2.4351e-02 6.7 0.00e+00 0.0 0.0e+00 0.0e+00 2.2e+01 0 0 0 0 5 0 0 0 0 5 0
> MatAssemblyEnd 20 1.0 1.3176e-01 1.0 0.00e+00 0.0 5.6e+02 2.1e+03 7.2e+01 3 0 10 2 16 3 0 10 2 16 0
> MatGetRowIJ 1 1.0 1.1516e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> MatGetOrdering 1 1.0 4.1008e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0
> MatView 16 1.3 1.0209e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 3 0 0 0 0 3 0
> MatPtAP 4 1.0 6.4001e-01 1.0 4.06e+07 1.0 1.1e+03 1.7e+04 1.0e+02 16 15 21 30 22 16 15 21 30 22 507
MatPtAP dominates the setup time. For profiling, you could register a
stage (PetscLogStageRegister) and time the setup separately from the
solve.
> MatPtAPSymbolic 4 1.0 3.7003e-01 1.0 0.00e+00 0.0 7.2e+02 2.0e+04 6.0e+01 9 0 13 22 13 9 0 13 22 13 0
> MatPtAPNumeric 4 1.0 2.7004e-01 1.0 4.06e+07 1.0 4.2e+02 1.2e+04 4.0e+01 7 15 8 8 9 7 15 8 8 9 1202
> MatGetRedundant 1 1.0 7.9393e-04 1.0 0.00e+00 0.0 1.7e+02 7.1e+03 4.0e+00 0 0 3 2 1 0 0 3 2 1 0
> MatGetLocalMat 4 1.0 3.9521e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 1 0 0 0 2 1 0 0 0 2 0
> MatGetBrAoCol 4 1.0 1.7719e-02 1.0 0.00e+00 0.0 4.3e+02 2.7e+04 8.0e+00 0 0 8 18 2 0 0 8 18 2 0
> MatGetSymTrans 8 1.0 1.3007e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
> KSPSetUp 7 1.0 1.3097e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 5 0 0 0 0 5 0
> KSPSolve 2 1.0 1.0450e+00 1.0 2.04e+08 1.0 3.4e+03 1.2e+04 7.5e+01 26 77 62 60 16 26 77 62 60 16 1563
> PCSetUp 1 1.0 8.6248e-01 1.0 6.21e+07 1.0 1.9e+03 1.1e+04 3.2e+02 21 23 35 32 69 21 23 35 32 69 576
> PCApply 6 1.0 8.4384e-01 1.0 1.61e+08 1.0 3.2e+03 9.0e+03 4.8e+01 21 60 59 44 10 21 60 59 44 11 1523
Do you know why there are 6 PCApply events? With four iterations of the
Krylov method, there should be only 5 events. Oh, it looks like you do
two solves. Is one of those with a different system?
Recall that the old KSPSolve time was over 3.35 seconds.
> -log_summary -ksp_monitor -ksp_view -ksp_converged_reason -pc_type mg
> -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson
> -mg_levels_ksp_max_it 1 -pc_mg_type full
>
> 0 KSP Residual norm 3.654533581988e-05
> 1 KSP Residual norm 8.730776244351e-07
> 2 KSP Residual norm 3.474626061661e-08
> 3 KSP Residual norm 1.813665557493e-09
This converges slightly faster, but ends up not paying off.
> Time (sec): 4.261e+00 1.00012 4.261e+00
> Objects: 2.950e+02 1.00000 2.950e+02
> Flops: 3.322e+08 1.00000 3.322e+08 2.658e+09
> Flops/sec: 7.797e+07 1.00012 7.796e+07 6.237e+08
> MPI Messages: 1.442e+03 1.00000 1.442e+03 1.154e+04
> MPI Message Lengths: 1.018e+07 1.00000 7.057e+03 8.141e+07
> MPI Reductions: 5.460e+02 1.00000
More messages, more work, etc., so not better.
> KSPSolve 2 1.0 1.2287e+00 1.0 2.70e+08 1.0 9.5e+03 5.8e+03 1.6e+02 29 81 82 68 30 29 81 82 68 30 1758
> PCSetUp 1 1.0 8.6414e-01 1.0 6.21e+07 1.0 1.9e+03 1.1e+04 3.2e+02 20 19 17 26 58 20 19 17 26 58 575
> PCApply 5 1.0 1.0571e+00 1.0 2.33e+08 1.0 9.3e+03 4.9e+03 1.4e+02 24 70 81 56 26 24 70 81 56 26 1764
It's still entirely possible that you can make Full MG beat V-cycles,
especially if you only need to converge up to discretization error. By
my figures, your good solver takes 12 work units to converge well below
discretization error (after Galerkin setup, but maybe you only need to
do that once?). If you only need to equal truncation error, this can be
brought down to about 5 (probably at best a 2x speedup in parallel).
This would involve a high-order (cubic) FMG prolongation.
Alternatively, you can speed up the implementation (and significantly
reduce memory usage) by creating geometric coarse levels and a
matrix-free implementation of MatSOR and MatMult. (The matrices are
great for experimenting, but if this solver is mission critical and
still a bottleneck, the matrix is an inefficient way to represent the
operator since it has very low arithmetic intensity/requires a lot of
memory bandwidth.) I predict you can probably speed up the solve by
perhaps another factor of 2 with a good matrix-free FMG implementation.
Do you want to go down this path?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130813/a4511d11/attachment.pgp>
More information about the petsc-users
mailing list