Setting tolerances for multigrid

Sun Jun 17 11:33:43 CDT 2007

  Switching this to petsc-amint

   Something is funky here. The results should be the same in both cases.
They are both using gmres and lu, on the same problem. The linear systems
should be solved in a single iteration in both cases (with lu as a direct
solve). When the -pc_type lu is used then the multigrid is NOT used, just
LU. Are you using matrix-free anywhere? At the start of the third linear
solve the residual norms are slightly different. This means something
else in your code (not in the linear solve) is doing something different
in the two cases. Is the other part of the code using nlevels somewhere?

   Barry

On Sun, 17 Jun 2007, Knut Erik Teigen wrote:

> Ok, I've attached two files, one with
> ./run -dmmg_galerkin -dmmg_nlevels 1 -ksp_type gmres -pc_type lu
> -ksp_truemonitor -ksp_converged_reason > lev1 
> 
> and one with
> 
> ./run -dmmg_galerkin -dmmg_nlevels 2 -ksp_type gmres -pc_type lu
> -mg_levels_1_ksp_type gmres -mg_levels_1_pc_type lu
> -ksp_converged_reason -ksp_truemonitor > lev2
> 
> The relative residual is 1.e-12, and I am using
> KSPSetInitialGuessNonzero.
> I've also printed the solution, and if you diff the two files, you can
> see that there is a large difference in the third step, e.g. 71.7852 vs.
> 70.9845. This shouldn't happen, even with a true residual of 1.e-5?
> 
> -Knut Erik-
> 
> On Sat, 2007-06-16 at 16:56 -0500, Barry Smith wrote:
> > 
> > On Sat, 16 Jun 2007, Knut Erik Teigen wrote:
> > 
> > > On Fri, 2007-06-15 at 14:30 -0500, Barry Smith wrote:
> > > >   What happens if you run the two level with -ksp_rtol 1.e-12
> > > 
> > > The exact same thing happens. At the third time step, the solution with
> > > two levels begin to deviate from the one with one level.
> > > 
> > > > 
> > > >   BTW: when you just change -dmmg_nlevels from 1 to 2 you are now
> > > > solving a NEW problem that has twice as many points in each direction,
> > > > this can also effect the time-stepping. You can use -dmmg_nlevels 2
> > > > -pc_type lu to compare with the -dmmg_nlevels 2
> > > 
> > > I forgot to mention that I adjust the coarsest level according to the
> > > number of levels, so that the finest level always contains the same
> > > amount of points. I.e. with one level it is 65x129, with two 33x65 and
> > > so on.
> > > 
> > > > 
> > > >   Always verify things with a direct solver before using multigrid
> > > > or any other iterative linear solver. Multigrid is ONLY an optimization
> > > > (faster, less memory), debug BEFORE turning on the optimizations.
> > > 
> > > Yes, I've already done this. With a direct solver, I get the same
> > > solution as with one level. Since multigrid is only an optimization as
> > > you mention, I don't understand why the solution should be different
> > > when running with two or more levels as opposed to only one.
> > > Here's the output when running with -pc_type lu. As you can see, with
> > > two levels, suddenly more iterations are needed for the third step.
> > > 
> > > -dmmg_nlevels 1
> > >  Start of time loop:
> > >  L-s reinit converged after   10 steps with norm  1.716E-09
> > >   0 KSP preconditioned resid norm 1.366287028032e+04 true resid norm 3.168341703825e+08 ||Ae||/||Ax|| 1.000000000000e+00
> > >   1 KSP preconditioned resid norm 3.325862387561e+02 true resid norm 8.783238319618e-07 ||Ae||/||Ax|| 2.772187832207e-15
> > >   2 KSP preconditioned resid norm 4.237324220037e-10 true resid norm 9.365968491320e-07 ||Ae||/||Ax|| 2.956110598807e-15
> > > Linear solve converged due to CONVERGED_RTOL
> >     0 KSP preconditioned resid norm 1.366743784848e+04 true resid norm 3.168385551957e+08 ||Ae||/||Ax|| 1.000000000000e+00
> > >   1 KSP preconditioned resid norm 3.578541063222e+02 true resid norm 3.047639475331e-06 ||Ae||/||Ax|| 9.618903461571e-15
> > >   2 KSP preconditioned resid norm 2.490853654289e-10 true resid norm 2.047397542219e-06 ||Ae||/||Ax|| 6.461958333808e-15
> > > Linear solve converged due to CONVERGED_RTOL
> > >   0 KSP preconditioned resid norm 1.373641039770e+04 true resid norm 4.083933364397e+08 ||Ae||/||Ax|| 1.000000000000e+00
> > >   1 KSP preconditioned resid norm 2.756300642422e+02 true resid norm 1.333942531534e-06 ||Ae||/||Ax|| 3.266318062786e-15
> > >   2 KSP preconditioned resid norm 1.967864218276e-09 true resid norm 4.708778143639e-06 ||Ae||/||Ax|| 1.153000728339e-14
> > > Linear solve converged due to CONVERGED_RTOL
> > > 
> > > -dmmg_nlevels 2
> > >  Start of time loop:
> > >  L-s reinit converged after   10 steps with norm  1.716E-09
> > >   0 KSP preconditioned resid norm 1.366287028032e+04 true resid norm 3.168341703825e+08 ||Ae||/||Ax|| 1.000000000000e+00
> > >   1 KSP preconditioned resid norm 3.325862387561e+02 true resid norm 8.783238319618e-07 ||Ae||/||Ax|| 2.772187832207e-15
> > >   2 KSP preconditioned resid norm 4.237324220037e-10 true resid norm 9.365968491320e-07 ||Ae||/||Ax|| 2.956110598807e-15
> > > Linear solve converged due to CONVERGED_RTOL
> > >   0 KSP preconditioned resid norm 1.366743784848e+04 true resid norm 3.168385551957e+08 ||Ae||/||Ax|| 1.000000000000e+00
> > >   1 KSP preconditioned resid norm 3.578541063222e+02 true resid norm 3.047639475331e-06 ||Ae||/||Ax|| 9.618903461571e-15
> > >   2 KSP preconditioned resid norm 2.490853654289e-10 true resid norm 2.047397542219e-06 ||Ae||/||Ax|| 6.461958333808e-15
> > > Linear solve converged due to CONVERGED_RTOL
> > >   0 KSP preconditioned resid norm 1.373641039770e+04 true resid norm 4.083933364397e+08 ||Ae||/||Ax|| 1.000000000000e+00
> > >   1 KSP preconditioned resid norm 2.826990181167e+02 true resid norm 3.621921471554e+07 ||Ae||/||Ax|| 8.868708542429e-02
> > >   2 KSP preconditioned resid norm 2.151728459050e+02 true resid norm 5.126312558070e+05 ||Ae||/||Ax|| 1.255239030774e-03
> > >   3 KSP preconditioned resid norm 9.420491033127e+01 true resid norm 7.221320422298e+02 ||Ae||/||Ax|| 1.768226799500e-06
> > > (...)
> > > 118 KSP preconditioned resid norm 2.011348229632e-04 true resid norm 1.933530019765e-05 ||Ae||/||Ax|| 4.734479843920e-14
> > > 119 KSP preconditioned resid norm 1.002113384828e-04 true resid norm 1.933774400466e-05 ||Ae||/||Ax|| 4.735078239337e-14
> > > 120 KSP preconditioned resid norm 3.976002405523e-07 true resid norm 1.947236185243e-05 ||Ae||/||Ax|| 4.768041031762e-14
> > > Linear solve converged due to CONVERGED_RTOL
> > > 
> > > 
> > 
> >   The true residuals both me, they are very big at "convergence" 1.e-5? Are you using KSPSetInitialGuessNonzero() and
> > using the previous solution as initial guess?
> > 
> >   Run with -ksp_type gmres -pc_type lu -ksp_monitor_true_residual. I bet the converged true residual is
> > much better. Send the output.
> > 
> >    Barry
> > 
> > 
> > > 
> > > > 
> > > > 
> > > >    Barry
> > > > 
> > > > On Fri, 15 Jun 2007, Knut Erik Teigen wrote:
> > > > 
> > > > > When running with those options I get this after the first timestep for
> > > > > both one and two levels:
> > > > > 159 KSP preconditioned resid norm 1.397494864058e-07 true resid norm
> > > > > 4.996893774970e-02 ||Ae||/||Ax|| 1.577132216812e-10
> > > > > Linear solve converged due to CONVERGED_RTOL
> > > > > 
> > > > > However, after the second timestep they start to get different.
> > > > > With two levels the norm is actually lower, but still, the computation
> > > > > diverges after about 6 time steps. With one level the computation runs
> > > > > smoothly. Here are the numbers:
> > > > > 
> > > > > -dmmg_nlevels 1
> > > > >   0 KSP preconditioned resid norm 1.674785901465e+02 
> > > > > true resid norm 4.083933362528e+08 ||Ae||/||Ax|| 1.000000000000e+00
> > > > >   1 KSP preconditioned resid norm 6.256839347623e+01 
> > > > > true resid norm 4.220478752135e+07 ||Ae||/||Ax|| 1.033434774147e-01
> > > > > (...)
> > > > > 159 KSP preconditioned resid norm 1.318896741223e-07 
> > > > > true resid norm 5.104451971417e-02 ||Ae||/||Ax|| 1.249886204866e-10
> > > > > Linear solve converged due to CONVERGED_RTOL
> > > > > 
> > > > > -dmmg_nlevels 2
> > > > >   0 KSP preconditioned resid norm 1.674785901465e+02 
> > > > > true resid norm 4.083933362528e+08 ||Ae||/||Ax|| 1.000000000000e+00
> > > > >   1 KSP preconditioned resid norm 6.252455641723e+01 
> > > > > true resid norm 5.561545924132e+07 ||Ae||/||Ax|| 1.361811134129e-01
> > > > > (...) 
> > > > > 159 KSP preconditioned resid norm 1.305901659826e-07 
> > > > > true resid norm 4.297164043386e-02 ||Ae||/||Ax|| 1.052212086224e-10
> > > > > Linear solve converged due to CONVERGED_RTOL
> > > > > 
> > > > > -Knut Erik-
> > > > > 
> > > > > On Fri, 2007-06-15 at 10:10 -0500, Barry Smith wrote:
> > > > > >   You need to make sure the linear solvers are converging properly
> > > > > > before even worrying about time-stepping. Run with -ksp_monitor_true_residual
> > > > > > (-ksp_truemonitor on older versions of PETSc) and -ksp_converged_reason
> > > > > > 
> > > > > >    Barry
> > > > > > 
> > > > > > 
> > > > > > On Fri, 15 Jun 2007, Knut Erik Teigen wrote:
> > > > > > 
> > > > > > > The computation runs just fine with -dmmg_nlevels 1.
> > > > > > > I use a grid of 65x129, with three levels, so that the coarsest grid is
> > > > > > > 17x33. 
> > > > > > > I've diffed the solutions, and they are equal for the first two time
> > > > > > > steps, but then they start to differ somewhat. Here is an excerpt:
> > > > > > > -dmmg_nlevels 1:
> > > > > > > 65.0427492820557        65.0216231814457      64.9793945321386        
> > > > > > > -dmmg_nlevels 3:
> > > > > > > 65.0920864319067        65.0711253401585      65.0292265582817
> > > > > > > 
> > > > > > > Using 2 or 4 levels produce the same result as 3 levels.
> > > > > > > 
> > > > > > > -Knut Erik-
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > On Fri, 2007-06-15 at 10:21 +0200, Matthew Knepley wrote:
> > > > > > > > This does not sound right to me. I think something must be
> > > > > > > > configured incorrectly. All MG does is accelerate the solution
> > > > > > > > of the fine grid problem. I think something must be wrong with
> > > > > > > > the specification here. It is easy to check. Run with -dmmg_nlevels 1
> > > > > > > > which just solves on the fine grid.
> > > > > > > > 
> > > > > > > >   Matt
> > > > > > > > 
> > > > > > > > On 6/15/07, Knut Erik Teigen <knutert at stud.ntnu.no> wrote:
> > > > > > > > > Thanks, Matthew. It seems like the tolerances weren't my problem,
> > > > > > > > > though. Even with a tolerance of 1e-10, the computation diverges after
> > > > > > > > > only a few time steps when using multigrid. With ordinary linear
> > > > > > > > > solvers, I can use a tolerance of 1e-4 and still get a satisfactory
> > > > > > > > > solution.
> > > > > > > > > Could it be that the Galerkin approximation for the coarser levels is
> > > > > > > > > too inaccurate for my problem?
> > > > > > > > >
> > > > > > > > > -Knut Erik-
> > > > > > > > >
> > > > > > > > > On Thu, 2007-06-14 at 11:32 -0500, Matthew Knepley wrote:
> > > > > > > > > > On 6/14/07, Knut Erik Teigen <knutert at stud.ntnu.no> wrote:
> > > > > > > > > > > Hello,
> > > > > > > > > > >
> > > > > > > > > > > How do I set the tolerances for the KSP solvers when using the
> > > > > > > > > > > DMMG routines without using command line arguments? It seems that
> > > > > > > > > > > I can use the argument -ksp_rtol on the command line, but in the code I
> > > > > > > > > > > can't use KSPSetTolerances since I don't have access to the KSP solver
> > > > > > > > > > > context?
> > > > > > > > > >
> > > > > > > > > > You can use DMMGGetKSP() for the finest level, and dmmg[level]->ksp for
> > > > > > > > > > any others.
> > > > > > > > > >
> > > > > > > > > > > Also, I see there is a reference to the function DMMGSetUseGalerkin
> > > > > > > > > > > in the source code, but I can't find the documentation for it.
> > > > > > > > > > > Does this function not exist yet? Again I would like to use
> > > > > > > > > > > -dmmg_galerkin, but setting it in the code instead of on the command
> > > > > > > > > > > line.
> > > > > > > > > >
> > > > > > > > > > I think you want this:
> > > > > > > > > >
> > > > > > > > > > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/DA/DMMGSetUseGalerkinCoarse.html
> > > > > > > > > >
> > > > > > > > > >   Matt
> > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Knut Erik Teigen
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > 
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > 
> > > 
> > 
> > 
> 