[petsc-users] Number of levels of multigrid : 2-3 is sufficient ??

[Barry Smith]

  Timothee,

     Thank you for reporting this issue, it is indeed disturbing and could be due to a performance regression we may have introduced by being too clever for our own good. Could you please rerun with the additional option -mg_levels_ksp_type richardson and send the same output?

   Thanks

  Barry

> On Oct 14, 2015, at 9:32 PM, Timothée Nicolas <timothee.nicolas at gmail.com> wrote:
> 
> Thank you Barry for pointing this out. Indeed on a system with no debugging the Jacobian evaluations no longer dominate the time (less than 10%). However the rest is similar, except the improvement from 2 to 3 levels is much better. Still it saturates after levels=3. I understand it in terms of CPU time thanks to Matthew's explanations, however what surprises me more is that KSP iterations are not more efficient. At the least, even if it takes more time to have more levels because of memory issues, I would expect KSP iterations to converge more rapidly with more levels, but it is not the case as you can see. Probably there is also a rationale behind this but I cannot see easily. 
> 
> I send the new outputs
> 
> Best
> 
> Timothee
> 
> 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> 1) Your timings are meaningless! You cannot compare timings when built with all debugging on, PERIOD!
> 
>   ##########################################################
>       #                                                        #
>       #                          WARNING!!!                    #
>       #                                                        #
>       #   This code was compiled with a debugging option,      #
>       #   To get timing results run ./configure                #
>       #   using --with-debugging=no, the performance will      #
>       #   be generally two or three times faster.              #
>       #                                                        #
>       ##########################################################
> 
> 2) Please run with -snes_view .
> 
> 3) Note that with 7 levels
> 
> SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
> 
> with 2 levels
> 
> SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
> 
> 
> The Jacobian evaluation is dominating the time! Likely if you fix the debugging this will be less the case
> 
>   Barry
> 
> > On Oct 13, 2015, at 9:23 PM, Timothée Nicolas <timothee.nicolas at gmail.com> wrote:
> >
> > Dear all,
> >
> > I have been playing around with multigrid recently, namely with /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and with my own implementation of a laplacian type problem. In all cases, I have noted no improvement whatsoever in the performance, whether in CPU time or KSP iteration, by varying the number of levels of the multigrid solver. As an example, I have attached the log_summary for ex5.c with nlevels = 2 to 7, launched by
> >
> > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9 -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor -log_summary
> >
> > where -pc_mg_levels is set to a number between 2 and 7.
> >
> > So there is a noticeable CPU time improvement from 2 levels to 3 levels (30%), and then no improvement whatsoever. I am surprised because with 6 levels of refinement of the DMDA the fine grid has more than 1200 points so with 3 levels the coarse grid still has more than 300 points which is still pretty large (I assume the ratio between grids is 2). I am wondering how the coarse solver efficiently solves the problem on the coarse grid with such a large number of points ? Given the principle of multigrid which is to erase the smooth part of the error with relaxation methods, which are usually efficient only for high frequency, I would expect optimal performance when the coarse grid is basically just a few points in each direction. Does anyone know why the performance saturates at low number of levels ? Basically what happens internally seems to be quite different from what I would expect...
> >
> > Best
> >
> > Timothee
> > <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
> 
> 
> <ex5_2_multigrid_levels.log><ex5_3_multigrid_levels.log><ex5_4_multigrid_levels.log><ex5_5_multigrid_levels.log><ex5_6_multigrid_levels.log><ex5_7_multigrid_levels.log>