[petsc-users] Issue using multi-grid as a pre-conditioner with KSP for a Poisson problem

[Dave May]

Please send your modified version of ex34. It will be faster to examine the
source and experiment with option choices locally rather than sending
emails back and forth.

Thanks,
  Dave

On Thu, 22 Jun 2017 at 03:13, Jason Lefley <jason.lefley at aclectic.com>
wrote:

> Hello,
>
> We are attempting to use the PETSc KSP solver framework in a fluid
> dynamics simulation we developed. The solution is part of a pressure
> projection and solves a Poisson problem. We use a cell-centered layout with
> a regular grid in 3d. We started with ex34.c from the KSP tutorials since
> it has the correct calls for the 3d DMDA, uses a cell-centered layout, and
> states that it works with multi-grid. We modified the operator construction
> function to match the coefficients and Dirichlet boundary conditions used
> in our problem (we’d also like to support Neumann but left those out for
> now to keep things simple). As a result of the modified boundary
> conditions, our version does not perform a null space removal on the right
> hand side or operator as the original did. We also modified the right hand
> side to contain a sinusoidal pattern for testing. Other than these changes,
> our code is the same as the original ex34.c
>
> With the default KSP options and using CG with the default pre-conditioner
> and without a pre-conditioner, we see good convergence. However, we’d like
> to accelerate the time to solution further and target larger problem sizes
> (>= 1024^3) if possible. Given these objectives, multi-grid as a
> pre-conditioner interests us. To understand the improvement that multi-grid
> provides, we ran ex45 from the KSP tutorials. ex34 with CG and no
> pre-conditioner appears to converge in a single iteration and we wanted to
> compare against a problem that has similar convergence patterns to our
> problem. Here’s the tests we ran with ex45:
>
> mpirun -n 16 ./ex45 -da_grid_x 129 -da_grid_y 129 -da_grid_z 129
>         time in KSPSolve(): 7.0178e+00
>         solver iterations: 157
>         KSP final norm of residual: 3.16874e-05
>
> mpirun -n 16 ./ex45 -da_grid_x 129 -da_grid_y 129 -da_grid_z 129 -ksp_type
> cg -pc_type none
>         time in KSPSolve(): 4.1072e+00
>         solver iterations: 213
>         KSP final norm of residual: 0.000138866
>
> mpirun -n 16 ./ex45 -da_grid_x 129 -da_grid_y 129 -da_grid_z 129 -ksp_type
> cg
>         time in KSPSolve(): 3.3962e+00
>         solver iterations: 88
>         KSP final norm of residual: 6.46242e-05
>
> mpirun -n 16 ./ex45 -da_grid_x 129 -da_grid_y 129 -da_grid_z 129 -pc_type
> mg -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1
> -mg_levels_pc_type bjacobi
>         time in KSPSolve(): 1.3201e+00
>         solver iterations: 4
>         KSP final norm of residual: 8.13339e-05
>
> mpirun -n 16 ./ex45 -da_grid_x 129 -da_grid_y 129 -da_grid_z 129 -pc_type
> mg -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1
> -mg_levels_pc_type bjacobi -ksp_type cg
>         time in KSPSolve(): 1.3008e+00
>         solver iterations: 4
>         KSP final norm of residual: 2.21474e-05
>
> We found the multi-grid pre-conditioner options in the KSP tutorials
> makefile. These results make sense; both the default GMRES and CG solvers
> converge and CG without a pre-conditioner takes more iterations. The
> multi-grid pre-conditioned runs are pretty dramatically accelerated and
> require only a handful of iterations.
>
> We ran our code (modified ex34.c as described above) with the same
> parameters:
>
> mpirun -n 16 ./solver_test -da_grid_x 128 -da_grid_y 128 -da_grid_z 128
>         time in KSPSolve(): 5.3729e+00
>         solver iterations: 123
>         KSP final norm of residual: 0.00595066
>
> mpirun -n 16 ./solver_test -da_grid_x 128 -da_grid_y 128 -da_grid_z 128
> -ksp_type cg -pc_type none
>         time in KSPSolve(): 3.6154e+00
>         solver iterations: 188
>         KSP final norm of residual: 0.00505943
>
> mpirun -n 16 ./solver_test -da_grid_x 128 -da_grid_y 128 -da_grid_z 128
> -ksp_type cg
>         time in KSPSolve(): 3.5661e+00
>         solver iterations: 98
>         KSP final norm of residual: 0.00967462
>
> mpirun -n 16 ./solver_test -da_grid_x 128 -da_grid_y 128 -da_grid_z 128
> -pc_type mg -pc_mg_levels 5 -mg_levels_ksp_type richardson
> -mg_levels_ksp_max_it 1 -mg_levels_pc_type bjacobi
>         time in KSPSolve(): 4.5606e+00
>         solver iterations: 44
>         KSP final norm of residual: 949.553
>
> mpirun -n 16 ./solver_test -da_grid_x 128 -da_grid_y 128 -da_grid_z 128
> -pc_type mg -pc_mg_levels 5 -mg_levels_ksp_type richardson
> -mg_levels_ksp_max_it 1 -mg_levels_pc_type bjacobi -ksp_type cg
>         time in KSPSolve(): 1.5481e+01
>         solver iterations: 198
>         KSP final norm of residual: 0.916558
>
> We performed all tests with petsc-3.7.6.
>
> The trends with CG and GMRES seem consistent with the results from ex45.
> However, with multi-grid, something doesn’t seem right. Convergence seems
> poor and the solves run for many more iterations than ex45 with multi-grid
> as a pre-conditioner. I extensively validated the code that builds the
> matrix and also confirmed that the solution produced by CG, when evaluated
> with the system of equations elsewhere in our simulation, produces the same
> residual as indicated by PETSc. Given that we only made minimal
> modifications to the original example code, it seems likely that the
> operators constructed for the multi-grid levels are ok.
>
> We also tried a variety of other suggested parameters for the multi-grid
> pre-conditioner as suggested in related mailing list posts but we didn’t
> observe any significant improvements over the results above.
>
> Is there anything we can do to check the validity of the coefficient
> matrices built for the different multi-grid levels? Does it look like there
> could be problems there? Or any other suggestions to achieve better results
> with multi-grid? I have the -log_view, -ksp_view, and convergence monitor
> output from the above tests and can post any of it if it would assist.
>
> Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20170622/2b9d265b/attachment.html>