[petsc-users] Number of levels of multigrid : 2-3 is sufficient ??

[Barry Smith]

> On Oct 15, 2015, at 12:26 AM, Timothée Nicolas <timothee.nicolas at gmail.com> wrote:
> 
> OK,
> 
> I ran an other battery of tests, here are the outputs. It seems to get a bit better when refining more as you suggested. For instance, for one more level of refinement, the CPU time saturation occurs for 5 levels instead of 3 previously. However the number of KSP iterations always tends to (marginally) increase with the number of levels. But in the same time, it always remain pretty low (less than 5 with extremely good convergence) so maybe it is not really surprising ?

  Yes, this is normal. You should not expect the number of outter KSP iterations to decrease with the number of levels, at best it will be the same. It will not decrease.

  Barry

> 
> Timothee
> 
> 2015-10-15 13:15 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> 
>   Wow, quick response! Yes the times still indicate that after 4 levels you get no improvement in time.
> 
> t = [1.5629e+01 , 6.2692e+00, 5.3451e+00, 5.4948e+00, 5.4940e+00, 5.7643e+00 ]
> 
> I'll look more specifically at the numbers to see where the time is being transformed tomorrow when I am less drunk. It is a trade off between the work saved in the direct solve vs the work needed for the coarser levels in the multigrid cycle.
> 
> Try refining the grid a couple more times, likely more levels will still help in that case
> 
>  Ahh, you should also try -pc_mg_type full
> 
> 
>   Barry
> 
> 
> 
> > On Oct 14, 2015, at 10:53 PM, Timothée Nicolas <timothee.nicolas at gmail.com> wrote:
> >
> > OK,
> >
> > Richardson is 30-70% faster for these tests, but other than this I don't see any change.
> >
> > Timothee
> >
> >
> >
> > 2015-10-15 12:37 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> >
> >   Timothee,
> >
> >      Thank you for reporting this issue, it is indeed disturbing and could be due to a performance regression we may have introduced by being too clever for our own good. Could you please rerun with the additional option -mg_levels_ksp_type richardson and send the same output?
> >
> >    Thanks
> >
> >   Barry
> >
> > > On Oct 14, 2015, at 9:32 PM, Timothée Nicolas <timothee.nicolas at gmail.com> wrote:
> > >
> > > Thank you Barry for pointing this out. Indeed on a system with no debugging the Jacobian evaluations no longer dominate the time (less than 10%). However the rest is similar, except the improvement from 2 to 3 levels is much better. Still it saturates after levels=3. I understand it in terms of CPU time thanks to Matthew's explanations, however what surprises me more is that KSP iterations are not more efficient. At the least, even if it takes more time to have more levels because of memory issues, I would expect KSP iterations to converge more rapidly with more levels, but it is not the case as you can see. Probably there is also a rationale behind this but I cannot see easily.
> > >
> > > I send the new outputs
> > >
> > > Best
> > >
> > > Timothee
> > >
> > > 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> > > 1) Your timings are meaningless! You cannot compare timings when built with all debugging on, PERIOD!
> > >
> > >   ##########################################################
> > >       #                                                        #
> > >       #                          WARNING!!!                    #
> > >       #                                                        #
> > >       #   This code was compiled with a debugging option,      #
> > >       #   To get timing results run ./configure                #
> > >       #   using --with-debugging=no, the performance will      #
> > >       #   be generally two or three times faster.              #
> > >       #                                                        #
> > >       ##########################################################
> > >
> > > 2) Please run with -snes_view .
> > >
> > > 3) Note that with 7 levels
> > >
> > > SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
> > >
> > > with 2 levels
> > >
> > > SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
> > >
> > >
> > > The Jacobian evaluation is dominating the time! Likely if you fix the debugging this will be less the case
> > >
> > >   Barry
> > >
> > > > On Oct 13, 2015, at 9:23 PM, Timothée Nicolas <timothee.nicolas at gmail.com> wrote:
> > > >
> > > > Dear all,
> > > >
> > > > I have been playing around with multigrid recently, namely with /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and with my own implementation of a laplacian type problem. In all cases, I have noted no improvement whatsoever in the performance, whether in CPU time or KSP iteration, by varying the number of levels of the multigrid solver. As an example, I have attached the log_summary for ex5.c with nlevels = 2 to 7, launched by
> > > >
> > > > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9 -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor -log_summary
> > > >
> > > > where -pc_mg_levels is set to a number between 2 and 7.
> > > >
> > > > So there is a noticeable CPU time improvement from 2 levels to 3 levels (30%), and then no improvement whatsoever. I am surprised because with 6 levels of refinement of the DMDA the fine grid has more than 1200 points so with 3 levels the coarse grid still has more than 300 points which is still pretty large (I assume the ratio between grids is 2). I am wondering how the coarse solver efficiently solves the problem on the coarse grid with such a large number of points ? Given the principle of multigrid which is to erase the smooth part of the error with relaxation methods, which are usually efficient only for high frequency, I would expect optimal performance when the coarse grid is basically just a few points in each direction. Does anyone know why the performance saturates at low number of levels ? Basically what happens internally seems to be quite different from what I would expect...
> > > >
> > > > Best
> > > >
> > > > Timothee
> > > > <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
> > >
> > >
> > > <ex5_2_multigrid_levels.log><ex5_3_multigrid_levels.log><ex5_4_multigrid_levels.log><ex5_5_multigrid_levels.log><ex5_6_multigrid_levels.log><ex5_7_multigrid_levels.log>
> >
> >
> > <ex5_2_multigrid_levels_richardson.log><ex5_3_multigrid_levels_richardson.log><ex5_4_multigrid_levels_richardson.log><ex5_5_multigrid_levels_richardson.log><ex5_6_multigrid_levels_richardson.log><ex5_7_multigrid_levels_richardson.log>
> 
> 
> <ex5_2_multigrid_levels_richardson_7_refine.log><ex5_3_multigrid_levels_richardson_7_refine.log><ex5_3_multigrid_levels_richardson_8_refine.log><ex5_4_multigrid_levels_richardson_7_refine.log><ex5_4_multigrid_levels_richardson_8_refine.log><ex5_5_multigrid_levels_richardson_7_refine.log><ex5_5_multigrid_levels_richardson_8_refine.log><ex5_6_multigrid_levels_richardson_7_refine.log><ex5_6_multigrid_levels_richardson_8_refine.log><ex5_7_multigrid_levels_richardson_7_refine.log><ex5_7_multigrid_levels_richardson_8_refine.log><ex5_8_multigrid_levels_richardson_7_refine.log><ex5_8_multigrid_levels_richardson_8_refine.log><ex5_pc_mg_full_2_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_3_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_4_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_5_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_6_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_7_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_8_multigrid_levels_richardson_7_refine.log>