[petsc-users] Number of levels of multigrid : 2-3 is sufficient ??
Timothée Nicolas
timothee.nicolas at gmail.com
Thu Oct 15 00:26:51 CDT 2015
OK,
I ran an other battery of tests, here are the outputs. It seems to get a
bit better when refining more as you suggested. For instance, for one more
level of refinement, the CPU time saturation occurs for 5 levels instead of
3 previously. However the number of KSP iterations always tends to
(marginally) increase with the number of levels. But in the same time, it
always remain pretty low (less than 5 with extremely good convergence) so
maybe it is not really surprising ?
Timothee
2015-10-15 13:15 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
>
> Wow, quick response! Yes the times still indicate that after 4 levels
> you get no improvement in time.
>
> t = [1.5629e+01 , 6.2692e+00, 5.3451e+00, 5.4948e+00, 5.4940e+00,
> 5.7643e+00 ]
>
> I'll look more specifically at the numbers to see where the time is being
> transformed tomorrow when I am less drunk. It is a trade off between the
> work saved in the direct solve vs the work needed for the coarser levels in
> the multigrid cycle.
>
> Try refining the grid a couple more times, likely more levels will still
> help in that case
>
> Ahh, you should also try -pc_mg_type full
>
>
> Barry
>
>
>
> > On Oct 14, 2015, at 10:53 PM, Timothée Nicolas <
> timothee.nicolas at gmail.com> wrote:
> >
> > OK,
> >
> > Richardson is 30-70% faster for these tests, but other than this I don't
> see any change.
> >
> > Timothee
> >
> >
> >
> > 2015-10-15 12:37 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> >
> > Timothee,
> >
> > Thank you for reporting this issue, it is indeed disturbing and
> could be due to a performance regression we may have introduced by being
> too clever for our own good. Could you please rerun with the additional
> option -mg_levels_ksp_type richardson and send the same output?
> >
> > Thanks
> >
> > Barry
> >
> > > On Oct 14, 2015, at 9:32 PM, Timothée Nicolas <
> timothee.nicolas at gmail.com> wrote:
> > >
> > > Thank you Barry for pointing this out. Indeed on a system with no
> debugging the Jacobian evaluations no longer dominate the time (less than
> 10%). However the rest is similar, except the improvement from 2 to 3
> levels is much better. Still it saturates after levels=3. I understand it
> in terms of CPU time thanks to Matthew's explanations, however what
> surprises me more is that KSP iterations are not more efficient. At the
> least, even if it takes more time to have more levels because of memory
> issues, I would expect KSP iterations to converge more rapidly with more
> levels, but it is not the case as you can see. Probably there is also a
> rationale behind this but I cannot see easily.
> > >
> > > I send the new outputs
> > >
> > > Best
> > >
> > > Timothee
> > >
> > > 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> > > 1) Your timings are meaningless! You cannot compare timings when built
> with all debugging on, PERIOD!
> > >
> > > ##########################################################
> > > # #
> > > # WARNING!!! #
> > > # #
> > > # This code was compiled with a debugging option, #
> > > # To get timing results run ./configure #
> > > # using --with-debugging=no, the performance will #
> > > # be generally two or three times faster. #
> > > # #
> > > ##########################################################
> > >
> > > 2) Please run with -snes_view .
> > >
> > > 3) Note that with 7 levels
> > >
> > > SNESJacobianEval 21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00
> 0.0e+00 0.0e+00 54 0 0 0 0 54 0 0 0 0 0
> > >
> > > with 2 levels
> > >
> > > SNESJacobianEval 6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00
> 0.0e+00 0.0e+00 34 0 0 0 0 34 0 0 0 0 0
> > >
> > >
> > > The Jacobian evaluation is dominating the time! Likely if you fix the
> debugging this will be less the case
> > >
> > > Barry
> > >
> > > > On Oct 13, 2015, at 9:23 PM, Timothée Nicolas <
> timothee.nicolas at gmail.com> wrote:
> > > >
> > > > Dear all,
> > > >
> > > > I have been playing around with multigrid recently, namely with
> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
> with my own implementation of a laplacian type problem. In all cases, I
> have noted no improvement whatsoever in the performance, whether in CPU
> time or KSP iteration, by varying the number of levels of the multigrid
> solver. As an example, I have attached the log_summary for ex5.c with
> nlevels = 2 to 7, launched by
> > > >
> > > > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9
> -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor
> -log_summary
> > > >
> > > > where -pc_mg_levels is set to a number between 2 and 7.
> > > >
> > > > So there is a noticeable CPU time improvement from 2 levels to 3
> levels (30%), and then no improvement whatsoever. I am surprised because
> with 6 levels of refinement of the DMDA the fine grid has more than 1200
> points so with 3 levels the coarse grid still has more than 300 points
> which is still pretty large (I assume the ratio between grids is 2). I am
> wondering how the coarse solver efficiently solves the problem on the
> coarse grid with such a large number of points ? Given the principle of
> multigrid which is to erase the smooth part of the error with relaxation
> methods, which are usually efficient only for high frequency, I would
> expect optimal performance when the coarse grid is basically just a few
> points in each direction. Does anyone know why the performance saturates at
> low number of levels ? Basically what happens internally seems to be quite
> different from what I would expect...
> > > >
> > > > Best
> > > >
> > > > Timothee
> > > >
> <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
> > >
> > >
> > >
> <ex5_2_multigrid_levels.log><ex5_3_multigrid_levels.log><ex5_4_multigrid_levels.log><ex5_5_multigrid_levels.log><ex5_6_multigrid_levels.log><ex5_7_multigrid_levels.log>
> >
> >
> >
> <ex5_2_multigrid_levels_richardson.log><ex5_3_multigrid_levels_richardson.log><ex5_4_multigrid_levels_richardson.log><ex5_5_multigrid_levels_richardson.log><ex5_6_multigrid_levels_richardson.log><ex5_7_multigrid_levels_richardson.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_2_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 20100 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0020.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_3_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 21027 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0021.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_3_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 21027 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0022.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_4_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 22071 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0023.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_4_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 21998 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0024.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_5_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 22986 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0025.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_5_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 22997 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0026.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_6_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 23963 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0027.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_6_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 23923 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0028.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_7_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 24882 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0029.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_7_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 24899 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0030.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_8_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 25830 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0031.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_8_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 25819 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0032.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_2_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 20024 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0033.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_3_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 20961 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0034.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_4_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 21872 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0035.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_5_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 22787 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0036.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_6_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 23686 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0037.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_7_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 24595 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0038.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_8_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 25490 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0039.obj>
More information about the petsc-users
mailing list