[petsc-users] HYPRE hanging or slow? from observation
Hao Zhang
hbcbh1999 at gmail.com
Sat Oct 21 23:16:52 CDT 2017
the reason is when I do finer grid simulation, matrix become more stiff.
Much larger condition number. just to give you a perspective, it will take
6000 iterations to converge and the solver does converge. I want to reduce
the number of iterations while keeping the convergence rate. that's main
drive to do so much heavy lifting around. please advise. snippet will be
provided upon request.
Thanks again.
On Sun, Oct 22, 2017 at 12:08 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> Oh, you change KSP but not hypre. I did not understand this.
>
> Why not just use GMRES all the time? Why mess with BCGS if it is not
> robust? Not worth the small optimization if it breaks everything.
>
> Barry
>
>
>
> > On Oct 21, 2017, at 11:05 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> >
> > this is the initial pressure solver output regarding use of PETSc. it
> failed to converge after 40000 iterations, then use GMRES.
> >
> > 39987 KSP preconditioned resid norm 3.853125986269e-08 true resid norm
> 1.147359212216e-05 ||r(i)||/||b|| 1.557696568706e-06
> > 39988 KSP preconditioned resid norm 3.853126044003e-08 true resid norm
> 1.147359257282e-05 ||r(i)||/||b|| 1.557696629889e-06
> > 39989 KSP preconditioned resid norm 3.853126052100e-08 true resid norm
> 1.147359233695e-05 ||r(i)||/||b|| 1.557696597866e-06
> > 39990 KSP preconditioned resid norm 3.853126027357e-08 true resid norm
> 1.147359219860e-05 ||r(i)||/||b|| 1.557696579083e-06
> > 39991 KSP preconditioned resid norm 3.853126058478e-08 true resid norm
> 1.147359234281e-05 ||r(i)||/||b|| 1.557696598662e-06
> > 39992 KSP preconditioned resid norm 3.853126064006e-08 true resid norm
> 1.147359261420e-05 ||r(i)||/||b|| 1.557696635506e-06
> > 39993 KSP preconditioned resid norm 3.853126050203e-08 true resid norm
> 1.147359235972e-05 ||r(i)||/||b|| 1.557696600957e-06
> > 39994 KSP preconditioned resid norm 3.853126050182e-08 true resid norm
> 1.147359253713e-05 ||r(i)||/||b|| 1.557696625043e-06
> > 39995 KSP preconditioned resid norm 3.853125976795e-08 true resid norm
> 1.147359226222e-05 ||r(i)||/||b|| 1.557696587720e-06
> > 39996 KSP preconditioned resid norm 3.853125805127e-08 true resid norm
> 1.147359262747e-05 ||r(i)||/||b|| 1.557696637308e-06
> > 39997 KSP preconditioned resid norm 3.853125811756e-08 true resid norm
> 1.147359216008e-05 ||r(i)||/||b|| 1.557696573853e-06
> > 39998 KSP preconditioned resid norm 3.853125827833e-08 true resid norm
> 1.147359238372e-05 ||r(i)||/||b|| 1.557696604216e-06
> > 39999 KSP preconditioned resid norm 3.853127937068e-08 true resid norm
> 1.147359264043e-05 ||r(i)||/||b|| 1.557696639067e-06
> > 40000 KSP preconditioned resid norm 3.853122732867e-08 true resid norm
> 1.147359257776e-05 ||r(i)||/||b|| 1.557696630559e-06
> > Linear solve did not converge due to DIVERGED_ITS iterations 40000
> > KSP Object: 24 MPI processes
> > type: bcgs
> > maximum iterations=40000, initial guess is zero
> > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > left preconditioning
> > using PRECONDITIONED norm type for convergence test
> > PC Object: 24 MPI processes
> > type: hypre
> > HYPRE BoomerAMG preconditioning
> > Cycle type V
> > Maximum number of levels 25
> > Maximum number of iterations PER hypre call 1
> > Convergence tolerance PER hypre call 0.
> > Threshold for strong coupling 0.25
> > Interpolation truncation factor 0.
> > Interpolation: max elements per row 0
> > Number of levels of aggressive coarsening 0
> > Number of paths for aggressive coarsening 1
> > Maximum row sums 0.9
> > Sweeps down 1
> > Sweeps up 1
> > Sweeps on coarse 1
> > Relax down symmetric-SOR/Jacobi
> > Relax up symmetric-SOR/Jacobi
> > Relax on coarse Gaussian-elimination
> > Relax weight (all) 1.
> > Outer relax weight (all) 1.
> > Using CF-relaxation
> > Not using more complex smoothers.
> > Measure type local
> > Coarsen type Falgout
> > Interpolation type classical
> > linear system matrix = precond matrix:
> > Mat Object: A 24 MPI processes
> > type: mpiaij
> > rows=497664, cols=497664
> > total: nonzeros=3363552, allocated nonzeros=6967296
> > total number of mallocs used during MatSetValues calls =0
> > has attached null space
> > not using I-node (on process 0) routines
> >
> > The solution diverges for p0! The residual is 3.853123e-08. Solve again
> using GMRES!
> > KSP Object: 24 MPI processes
> > type: gmres
> > restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > happy breakdown tolerance 1e-30
> > maximum iterations=40000, initial guess is zero
> > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > left preconditioning
> > using PRECONDITIONED norm type for convergence test
> > PC Object: 24 MPI processes
> > type: hypre
> > HYPRE BoomerAMG preconditioning
> > Cycle type V
> > Maximum number of levels 25
> > Maximum number of iterations PER hypre call 1
> > Convergence tolerance PER hypre call 0.
> > Threshold for strong coupling 0.25
> > Interpolation truncation factor 0.
> > Interpolation: max elements per row 0
> > Number of levels of aggressive coarsening 0
> > Number of paths for aggressive coarsening 1
> > Maximum row sums 0.9
> > Sweeps down 1
> > Sweeps up 1
> > Sweeps on coarse 1
> > Relax down symmetric-SOR/Jacobi
> > Relax up symmetric-SOR/Jacobi
> > Relax on coarse Gaussian-elimination
> > Relax weight (all) 1.
> > Outer relax weight (all) 1.
> > Using CF-relaxation
> > Not using more complex smoothers.
> > Measure type local
> > Coarsen type Falgout
> > Interpolation type classical
> > linear system matrix = precond matrix:
> > Mat Object: A 24 MPI processes
> > type: mpiaij
> > rows=497664, cols=497664
> > total: nonzeros=3363552, allocated nonzeros=6967296
> > total number of mallocs used during MatSetValues calls =0
> > has attached null space
> > not using I-node (on process 0) routines
> > 0 KSP preconditioned resid norm 1.593802941804e+01 true resid norm
> 7.365742695119e+00 ||r(i)||/||b|| 1.000000000000e+00
> > 1 KSP preconditioned resid norm 6.338666661133e-01 true resid norm
> 2.880722209358e+00 ||r(i)||/||b|| 3.910973174867e-01
> > 2 KSP preconditioned resid norm 3.913420828350e-02 true resid norm
> 9.544089760671e-01 ||r(i)||/||b|| 1.295740315093e-01
> > 3 KSP preconditioned resid norm 2.928070366435e-03 true resid norm
> 1.874294004628e-01 ||r(i)||/||b|| 2.544609664236e-02
> > 4 KSP preconditioned resid norm 2.165607525823e-04 true resid norm
> 3.121122463949e-02 ||r(i)||/||b|| 4.237349298146e-03
> > 5 KSP preconditioned resid norm 1.635476480407e-05 true resid norm
> 3.984315313831e-03 ||r(i)||/||b|| 5.409251284967e-04
> > 6 KSP preconditioned resid norm 1.283358350575e-06 true resid norm
> 4.566583802915e-04 ||r(i)||/||b|| 6.199760149022e-05
> > 7 KSP preconditioned resid norm 8.479469225747e-08 true resid norm
> 3.824581791112e-05 ||r(i)||/||b|| 5.192391248810e-06
> > 8 KSP preconditioned resid norm 5.358636504683e-09 true resid norm
> 2.829730442033e-06 ||r(i)||/||b|| 3.841744898187e-07
> > 9 KSP preconditioned resid norm 3.447874504193e-10 true resid norm
> 1.756617036538e-07 ||r(i)||/||b|| 2.384847135242e-08
> > 10 KSP preconditioned resid norm 2.480228743414e-11 true resid norm
> 1.309399823577e-08 ||r(i)||/||b|| 1.777688792258e-09
> > 11 KSP preconditioned resid norm 1.728967759950e-12 true resid norm
> 9.406967045789e-10 ||r(i)||/||b|| 1.277124036931e-10
> > 12 KSP preconditioned resid norm 1.075458632828e-13 true resid norm
> 5.994505136212e-11 ||r(i)||/||b|| 8.138358050689e-12
> > Linear solve converged due to CONVERGED_RTOL iterations 12
> > KSP Object: 24 MPI processes
> > type: gmres
> > restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > happy breakdown tolerance 1e-30
> > maximum iterations=40000, initial guess is zero
> > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > left preconditioning
> > using PRECONDITIONED norm type for convergence test
> > PC Object: 24 MPI processes
> > type: hypre
> > HYPRE BoomerAMG preconditioning
> > Cycle type V
> > Maximum number of levels 25
> > Maximum number of iterations PER hypre call 1
> > Convergence tolerance PER hypre call 0.
> > Threshold for strong coupling 0.25
> > Interpolation truncation factor 0.
> > Interpolation: max elements per row 0
> > Number of levels of aggressive coarsening 0
> > Number of paths for aggressive coarsening 1
> > Maximum row sums 0.9
> > Sweeps down 1
> > Sweeps up 1
> > Sweeps on coarse 1
> > Relax down symmetric-SOR/Jacobi
> > Relax up symmetric-SOR/Jacobi
> > Relax on coarse Gaussian-elimination
> > Relax weight (all) 1.
> > Outer relax weight (all) 1.
> > Using CF-relaxation
> > Not using more complex smoothers.
> > Measure type local
> > Coarsen type Falgout
> > Interpolation type classical
> > linear system matrix = precond matrix:
> > Mat Object: A 24 MPI processes
> > type: mpiaij
> > rows=497664, cols=497664
> > total: nonzeros=3363552, allocated nonzeros=6967296
> > total number of mallocs used during MatSetValues calls =0
> > has attached null space
> > not using I-node (on process 0) routines
> > The max singular value of A = 1.000872 in poisson_solver3d_P0_vd
> > The min singular value of A = 0.667688 in poisson_solver3d_P0_vd
> > The Cond Num of A = 1.499012 in poisson_solver3d_P0_vd
> > In poisson_solver3d_pressure(): num_iter = 12, rel_residual =
> 1.075459e-13
> >
> > The max value of p0 is 0.03115845493408858
> >
> > The min value of p0 is -0.07156715468428149
> >
> > On Sun, Oct 22, 2017 at 12:00 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >
> > > On Oct 21, 2017, at 10:50 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > >
> > > the incompressible NS solver algorithm call PETSc solver at different
> stage of each time step. The one you were saying "This is good. 12 digit
> reduction" is after the initial pressure solver, in which usually HYPRE
> doesn't give a good convergence, so the fall-back solver GMRES will be
> called after.
> >
> > Hmm, I don't understand. hypre should do well on a pressure solve. In
> fact, very well.
> > >
> > > Barry, you were mentioning that I could have a wrong nullspace. that
> particular solver is aimed to give an initial pressure profile for 3d
> incompressible NS simulation using all neumann boundary conditions. could
> you give some insight how to test if I have a wrong nullspace etc?
> >
> > -ksp_test_null_space
> >
> > But if your null space is consistently from all Neumann boundary
> conditions then it likely is not wrong.
> >
> > Barry
> >
> > >
> > > Thanks!
> > >
> > > On Sat, Oct 21, 2017 at 11:41 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > > This is good. You get more than 12 digit reduction in the true
> residual norm. This is good AMG convergence. Expected when everything goes
> well.
> > >
> > > What is different in this case from the previous case that does not
> converge reasonably?
> > >
> > > Barry
> > >
> > >
> > > > On Oct 21, 2017, at 9:29 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > >
> > > > Barry, Please advise what you make of this? this is poisson solver
> with all neumann BC 3d case Finite difference Scheme was used.
> > > > Thanks! I'm in learning mode.
> > > >
> > > > KSP Object: 24 MPI processes
> > > > type: bcgs
> > > > maximum iterations=40000, initial guess is zero
> > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > left preconditioning
> > > > using PRECONDITIONED norm type for convergence test
> > > > PC Object: 24 MPI processes
> > > > type: hypre
> > > > HYPRE BoomerAMG preconditioning
> > > > Cycle type V
> > > > Maximum number of levels 25
> > > > Maximum number of iterations PER hypre call 1
> > > > Convergence tolerance PER hypre call 0.
> > > > Threshold for strong coupling 0.25
> > > > Interpolation truncation factor 0.
> > > > Interpolation: max elements per row 0
> > > > Number of levels of aggressive coarsening 0
> > > > Number of paths for aggressive coarsening 1
> > > > Maximum row sums 0.9
> > > > Sweeps down 1
> > > > Sweeps up 1
> > > > Sweeps on coarse 1
> > > > Relax down symmetric-SOR/Jacobi
> > > > Relax up symmetric-SOR/Jacobi
> > > > Relax on coarse Gaussian-elimination
> > > > Relax weight (all) 1.
> > > > Outer relax weight (all) 1.
> > > > Using CF-relaxation
> > > > Not using more complex smoothers.
> > > > Measure type local
> > > > Coarsen type Falgout
> > > > Interpolation type classical
> > > > linear system matrix = precond matrix:
> > > > Mat Object: A 24 MPI processes
> > > > type: mpiaij
> > > > rows=497664, cols=497664
> > > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > > total number of mallocs used during MatSetValues calls =0
> > > > has attached null space
> > > > not using I-node (on process 0) routines
> > > > 0 KSP preconditioned resid norm 2.697270170623e-02 true resid norm
> 9.637159071207e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > > 1 KSP preconditioned resid norm 4.828857674609e-04 true resid norm
> 6.294379664645e-01 ||r(i)||/||b|| 6.531364293291e-02
> > > > 2 KSP preconditioned resid norm 4.533649595815e-06 true resid norm
> 1.135508605857e-02 ||r(i)||/||b|| 1.178260727531e-03
> > > > 3 KSP preconditioned resid norm 1.131704082606e-07 true resid norm
> 1.196393029874e-04 ||r(i)||/||b|| 1.241437462051e-05
> > > > 4 KSP preconditioned resid norm 3.866281379569e-10 true resid norm
> 5.121520801846e-07 ||r(i)||/||b|| 5.314347064320e-08
> > > > 5 KSP preconditioned resid norm 1.070114785241e-11 true resid norm
> 1.203693733135e-08 ||r(i)||/||b|| 1.249013038221e-09
> > > > 6 KSP preconditioned resid norm 2.578780418765e-14 true resid norm
> 6.020297525927e-11 ||r(i)||/||b|| 6.246962908306e-12
> > > > 7 KSP preconditioned resid norm 8.691764190203e-16 true resid norm
> 1.866088098154e-12 ||r(i)||/||b|| 1.936346680973e-13
> > > > Linear solve converged due to CONVERGED_ATOL iterations 7
> > > >
> > > >
> > > >
> > > > On Sat, Oct 21, 2017 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > >
> > > > > On Oct 21, 2017, at 5:47 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > >
> > > > > hi, Barry:
> > > > > what do you mean absurd by setting tolerance =1e-14?
> > > >
> > > > Trying to decrease the initial residual norm down by a factor of
> 1e-14 with an iterative method (or even direct method) is unrealistic,
> usually unachievable) and almost never necessary. You are requiring || r_n
> || < 1.e-14 || r_0|| when with double precision numbers you only have
> roughly 14 decimal digits total to compute with. Round off alone will lead
> to differences far larger than 1e-14
> > > >
> > > > If you are using the solver in the context of a nonlinear problem
> (i.e. inside Newton's method) then 1.e-6 is generally more than plenty to
> get quadratic convergence of Newton's method.
> > > >
> > > > If you are solving a linear problem then it is extremely likely
> that errors due to discretization errors (from finite element method etc)
> and the model are much much larger than even 1.e-8.
> > > >
> > > > So, in summary
> > > >
> > > > 1.e-14 is probably unachievable
> > > >
> > > > 1.e-14 is almost for sure not needed.
> > > >
> > > > Barry
> > > >
> > > >
> > > > >
> > > > > On Sat, Oct 21, 2017 at 18:42 Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > >
> > > > > Run with -ksp_view_mat binary -ksp_view_rhs binary and send the
> resulting output file called binaryoutput to petsc-maint at mcs.anl.gov
> > > > >
> > > > > Note you can also use -ksp_type gmres with hypre, unlikely to be
> a reason to use bcgs
> > > > >
> > > > > BTW: tolerances: relative=1e-14, is absurd
> > > > >
> > > > > My guess is your null space is incorrect.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > On Oct 21, 2017, at 4:34 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > > >
> > > > > > if this solver doesn't converge. I have a fall-back solution,
> which uses GMRES solver. this setup is fine with me. I just want to know if
> HYPRE is a reliable solution for me. Or I will have to go without
> preconditioner.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > On Sat, Oct 21, 2017 at 5:30 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > > > this is serial run. still dumping output. parallel more or less
> the same.
> > > > > >
> > > > > > KSP Object: 1 MPI processes
> > > > > > type: bcgs
> > > > > > maximum iterations=40000, initial guess is zero
> > > > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > > > left preconditioning
> > > > > > using PRECONDITIONED norm type for convergence test
> > > > > > PC Object: 1 MPI processes
> > > > > > type: hypre
> > > > > > HYPRE BoomerAMG preconditioning
> > > > > > Cycle type V
> > > > > > Maximum number of levels 25
> > > > > > Maximum number of iterations PER hypre call 1
> > > > > > Convergence tolerance PER hypre call 0.
> > > > > > Threshold for strong coupling 0.25
> > > > > > Interpolation truncation factor 0.
> > > > > > Interpolation: max elements per row 0
> > > > > > Number of levels of aggressive coarsening 0
> > > > > > Number of paths for aggressive coarsening 1
> > > > > > Maximum row sums 0.9
> > > > > > Sweeps down 1
> > > > > > Sweeps up 1
> > > > > > Sweeps on coarse 1
> > > > > > Relax down symmetric-SOR/Jacobi
> > > > > > Relax up symmetric-SOR/Jacobi
> > > > > > Relax on coarse Gaussian-elimination
> > > > > > Relax weight (all) 1.
> > > > > > Outer relax weight (all) 1.
> > > > > > Using CF-relaxation
> > > > > > Not using more complex smoothers.
> > > > > > Measure type local
> > > > > > Coarsen type Falgout
> > > > > > Interpolation type classical
> > > > > > linear system matrix = precond matrix:
> > > > > > Mat Object: A 1 MPI processes
> > > > > > type: seqaij
> > > > > > rows=497664, cols=497664
> > > > > > total: nonzeros=3363552, allocated nonzeros=3483648
> > > > > > total number of mallocs used during MatSetValues calls =0
> > > > > > has attached null space
> > > > > > not using I-node routines
> > > > > > 0 KSP preconditioned resid norm 1.630377897956e+01 true resid
> norm 7.365742695123e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > > > > 1 KSP preconditioned resid norm 3.815819909205e-02 true resid
> norm 7.592300567353e-01 ||r(i)||/||b|| 1.030758320187e-01
> > > > > > 2 KSP preconditioned resid norm 4.312671277701e-04 true resid
> norm 2.553060965521e-02 ||r(i)||/||b|| 3.466128360975e-03
> > > > > > 3 KSP preconditioned resid norm 3.011875330569e-04 true resid
> norm 6.836208627386e-04 ||r(i)||/||b|| 9.281085303065e-05
> > > > > > 4 KSP preconditioned resid norm 3.011783821295e-04 true resid
> norm 2.504661140204e-04 ||r(i)||/||b|| 3.400418998972e-05
> > > > > > 5 KSP preconditioned resid norm 3.011783818372e-04 true resid
> norm 2.504053673004e-04 ||r(i)||/||b|| 3.399594279422e-05
> > > > > > 6 KSP preconditioned resid norm 3.011783818375e-04 true resid
> norm 2.503984078890e-04 ||r(i)||/||b|| 3.399499795925e-05
> > > > > > 7 KSP preconditioned resid norm 3.011783887442e-04 true resid
> norm 2.504121501772e-04 ||r(i)||/||b|| 3.399686366224e-05
> > > > > > 8 KSP preconditioned resid norm 3.010913654181e-04 true resid
> norm 2.504150259031e-04 ||r(i)||/||b|| 3.399725408124e-05
> > > > > > 9 KSP preconditioned resid norm 3.006520688232e-04 true resid
> norm 2.504061607382e-04 ||r(i)||/||b|| 3.399605051423e-05
> > > > > > 10 KSP preconditioned resid norm 3.007309991942e-04 true resid
> norm 2.503843638523e-04 ||r(i)||/||b|| 3.399309128978e-05
> > > > > > 11 KSP preconditioned resid norm 3.015946168077e-04 true resid
> norm 2.503644677844e-04 ||r(i)||/||b|| 3.399039012728e-05
> > > > > > 12 KSP preconditioned resid norm 2.956643907377e-04 true resid
> norm 2.503863046509e-04 ||r(i)||/||b|| 3.399335477965e-05
> > > > > > 13 KSP preconditioned resid norm 2.997992358936e-04 true resid
> norm 2.504336903093e-04 ||r(i)||/||b|| 3.399978802886e-05
> > > > > > 14 KSP preconditioned resid norm 2.481415420420e-05 true resid
> norm 2.491591201250e-04 ||r(i)||/||b|| 3.382674774806e-05
> > > > > > 15 KSP preconditioned resid norm 2.615494786181e-05 true resid
> norm 2.490353237273e-04 ||r(i)||/||b|| 3.380994069915e-05
> > > > > > 16 KSP preconditioned resid norm 2.645126692130e-05 true resid
> norm 2.490535523344e-04 ||r(i)||/||b|| 3.381241548111e-05
> > > > > > 17 KSP preconditioned resid norm 2.667223026209e-05 true resid
> norm 2.490482602536e-04 ||r(i)||/||b|| 3.381169700898e-05
> > > > > > 18 KSP preconditioned resid norm 2.650813432116e-05 true resid
> norm 2.490473169014e-04 ||r(i)||/||b|| 3.381156893606e-05
> > > > > > 19 KSP preconditioned resid norm 2.613309555449e-05 true resid
> norm 2.490465633690e-04 ||r(i)||/||b|| 3.381146663375e-05
> > > > > > 20 KSP preconditioned resid norm 2.644160446804e-05 true resid
> norm 2.490532739949e-04 ||r(i)||/||b|| 3.381237769272e-05
> > > > > > 21 KSP preconditioned resid norm 2.635987608975e-05 true resid
> norm 2.490499548926e-04 ||r(i)||/||b|| 3.381192707933e-05
> > > > > > 22 KSP preconditioned resid norm 2.640527129095e-05 true resid
> norm 2.490594066529e-04 ||r(i)||/||b|| 3.381321028466e-05
> > > > > > 23 KSP preconditioned resid norm 2.627505117691e-05 true resid
> norm 2.490550162585e-04 ||r(i)||/||b|| 3.381261422875e-05
> > > > > > 24 KSP preconditioned resid norm 2.642659196388e-05 true resid
> norm 2.490504347640e-04 ||r(i)||/||b|| 3.381199222842e-05
> > > > > > 25 KSP preconditioned resid norm 2.659432190695e-05 true resid
> norm 2.490510775152e-04 ||r(i)||/||b|| 3.381207949065e-05
> > > > > > 26 KSP preconditioned resid norm 2.687918062951e-05 true resid
> norm 2.490518882015e-04 ||r(i)||/||b|| 3.381218955237e-05
> > > > > > 27 KSP preconditioned resid norm 2.662909048432e-05 true resid
> norm 2.490446263285e-04 ||r(i)||/||b|| 3.381120365409e-05
> > > > > > 28 KSP preconditioned resid norm 2.085466483199e-05 true resid
> norm 2.490131612366e-04 ||r(i)||/||b|| 3.380693183886e-05
> > > > > > 29 KSP preconditioned resid norm 2.098541330282e-05 true resid
> norm 2.490126933398e-04 ||r(i)||/||b|| 3.380686831549e-05
> > > > > > 30 KSP preconditioned resid norm 2.175345180286e-05 true resid
> norm 2.490098852429e-04 ||r(i)||/||b|| 3.380648707805e-05
> > > > > > 31 KSP preconditioned resid norm 2.182182437676e-05 true resid
> norm 2.490028301020e-04 ||r(i)||/||b|| 3.380552924648e-05
> > > > > > 32 KSP preconditioned resid norm 2.152970404369e-05 true resid
> norm 2.490089939838e-04 ||r(i)||/||b|| 3.380636607747e-05
> > > > > > 33 KSP preconditioned resid norm 2.187932450016e-05 true resid
> norm 2.490085293931e-04 ||r(i)||/||b|| 3.380630300295e-05
> > > > > > 34 KSP preconditioned resid norm 2.207255875067e-05 true resid
> norm 2.490039036092e-04 ||r(i)||/||b|| 3.380567498971e-05
> > > > > > 35 KSP preconditioned resid norm 2.205060279701e-05 true resid
> norm 2.490101636150e-04 ||r(i)||/||b|| 3.380652487086e-05
> > > > > > 36 KSP preconditioned resid norm 2.168654200416e-05 true resid
> norm 2.490091609876e-04 ||r(i)||/||b|| 3.380638875052e-05
> > > > > > 37 KSP preconditioned resid norm 2.164521042361e-05 true resid
> norm 2.490083143913e-04 ||r(i)||/||b|| 3.380627381352e-05
> > > > > > 38 KSP preconditioned resid norm 2.154429063973e-05 true resid
> norm 2.490075485470e-04 ||r(i)||/||b|| 3.380616983972e-05
> > > > > > 39 KSP preconditioned resid norm 2.165962086228e-05 true resid
> norm 2.490099695056e-04 ||r(i)||/||b|| 3.380649851786e-05
> > > > > > 40 KSP preconditioned resid norm 2.153877616091e-05 true resid
> norm 2.490090652619e-04 ||r(i)||/||b|| 3.380637575444e-05
> > > > > > 41 KSP preconditioned resid norm 2.347651187611e-05 true resid
> norm 2.490233544624e-04 ||r(i)||/||b|| 3.380831570825e-05
> > > > > > 42 KSP preconditioned resid norm 2.352860162514e-05 true resid
> norm 2.490191394202e-04 ||r(i)||/||b|| 3.380774345879e-05
> > > > > > 43 KSP preconditioned resid norm 2.312377506928e-05 true resid
> norm 2.490209491359e-04 ||r(i)||/||b|| 3.380798915237e-05
> > > > > > 44 KSP preconditioned resid norm 2.295770973533e-05 true resid
> norm 2.490178136759e-04 ||r(i)||/||b|| 3.380756347093e-05
> > > > > > 45 KSP preconditioned resid norm 2.833646456041e-05 true resid
> norm 2.489991602651e-04 ||r(i)||/||b|| 3.380503101608e-05
> > > > > > 46 KSP preconditioned resid norm 2.760296424494e-05 true resid
> norm 2.490104320666e-04 ||r(i)||/||b|| 3.380656131682e-05
> > > > > > 47 KSP preconditioned resid norm 2.451504295239e-05 true resid
> norm 2.490241388672e-04 ||r(i)||/||b|| 3.380842220189e-05
> > > > > > 48 KSP preconditioned resid norm 2.512391514098e-05 true resid
> norm 2.490245923753e-04 ||r(i)||/||b|| 3.380848377180e-05
> > > > > > 49 KSP preconditioned resid norm 2.483419450528e-05 true resid
> norm 2.490273364402e-04 ||r(i)||/||b|| 3.380885631602e-05
> > > > > > 50 KSP preconditioned resid norm 2.507460538466e-05 true resid
> norm 2.490309488780e-04 ||r(i)||/||b|| 3.380934675371e-05
> > > > > > 51 KSP preconditioned resid norm 2.499708772881e-05 true resid
> norm 2.490300908170e-04 ||r(i)||/||b|| 3.380923026022e-05
> > > > > > 52 KSP preconditioned resid norm 1.059778259446e-05 true resid
> norm 2.489352833521e-04 ||r(i)||/||b|| 3.379635885420e-05
> > > > > > 53 KSP preconditioned resid norm 1.074975117060e-05 true resid
> norm 2.489294722901e-04 ||r(i)||/||b|| 3.379556992330e-05
> > > > > > 54 KSP preconditioned resid norm 1.095242219559e-05 true resid
> norm 2.489295454212e-04 ||r(i)||/||b|| 3.379557985184e-05
> > > > > > 55 KSP preconditioned resid norm 8.359999674720e-06 true resid
> norm 2.489673581944e-04 ||r(i)||/||b|| 3.380071345137e-05
> > > > > > 56 KSP preconditioned resid norm 8.368232998470e-06 true resid
> norm 2.489700421343e-04 ||r(i)||/||b|| 3.380107783281e-05
> > > > > > 57 KSP preconditioned resid norm 8.443378041101e-06 true resid
> norm 2.489702900875e-04 ||r(i)||/||b|| 3.380111149584e-05
> > > > > > 58 KSP preconditioned resid norm 8.647159584302e-06 true resid
> norm 2.489640805831e-04 ||r(i)||/||b|| 3.380026847095e-05
> > > > > > 59 KSP preconditioned resid norm 1.024742790737e-05 true resid
> norm 2.489447846660e-04 ||r(i)||/||b|| 3.379764878711e-05
> > > > > > 60 KSP preconditioned resid norm 1.033394118910e-05 true resid
> norm 2.489441404923e-04 ||r(i)||/||b|| 3.379756133175e-05
> > > > > > 61 KSP preconditioned resid norm 1.030066336446e-05 true resid
> norm 2.489399918556e-04 ||r(i)||/||b|| 3.379699809776e-05
> > > > > > 62 KSP preconditioned resid norm 1.029956398963e-05 true resid
> norm 2.489445295139e-04 ||r(i)||/||b|| 3.379761414674e-05
> > > > > > 63 KSP preconditioned resid norm 1.028190129002e-05 true resid
> norm 2.489456200527e-04 ||r(i)||/||b|| 3.379776220225e-05
> > > > > > 64 KSP preconditioned resid norm 9.878799185773e-06 true resid
> norm 2.489488742330e-04 ||r(i)||/||b|| 3.379820400160e-05
> > > > > > 65 KSP preconditioned resid norm 9.917711104174e-06 true resid
> norm 2.489478066593e-04 ||r(i)||/||b|| 3.379805906391e-05
> > > > > > 66 KSP preconditioned resid norm 1.003572019576e-05 true resid
> norm 2.489441995703e-04 ||r(i)||/||b|| 3.379756935240e-05
> > > > > > 67 KSP preconditioned resid norm 9.924487278236e-06 true resid
> norm 2.489475403451e-04 ||r(i)||/||b|| 3.379802290812e-05
> > > > > > 68 KSP preconditioned resid norm 9.804213483359e-06 true resid
> norm 2.489457781760e-04 ||r(i)||/||b|| 3.379778366964e-05
> > > > > > 69 KSP preconditioned resid norm 9.748922705476e-06 true resid
> norm 2.489408473578e-04 ||r(i)||/||b|| 3.379711424383e-05
> > > > > > 70 KSP preconditioned resid norm 9.886044523689e-06 true resid
> norm 2.489514438395e-04 ||r(i)||/||b|| 3.379855286071e-05
> > > > > > 71 KSP preconditioned resid norm 1.083888478689e-05 true resid
> norm 2.489420898851e-04 ||r(i)||/||b|| 3.379728293386e-05
> > > > > > 72 KSP preconditioned resid norm 1.106561823757e-05 true resid
> norm 2.489364778104e-04 ||r(i)||/||b|| 3.379652101821e-05
> > > > > > 73 KSP preconditioned resid norm 1.132091515426e-05 true resid
> norm 2.489456804535e-04 ||r(i)||/||b|| 3.379777040248e-05
> > > > > > 74 KSP preconditioned resid norm 1.330905328963e-05 true resid
> norm 2.489317458981e-04 ||r(i)||/||b|| 3.379587859660e-05
> > > > > > 75 KSP preconditioned resid norm 1.305555302619e-05 true resid
> norm 2.489320939810e-04 ||r(i)||/||b|| 3.379592585359e-05
> > > > > > 76 KSP preconditioned resid norm 1.308083397399e-05 true resid
> norm 2.489299951581e-04 ||r(i)||/||b|| 3.379564090977e-05
> > > > > > 77 KSP preconditioned resid norm 1.320098861853e-05 true resid
> norm 2.489323669317e-04 ||r(i)||/||b|| 3.379596291036e-05
> > > > > > 78 KSP preconditioned resid norm 1.300160788274e-05 true resid
> norm 2.489306393356e-04 ||r(i)||/||b|| 3.379572836564e-05
> > > > > > 79 KSP preconditioned resid norm 1.317651537793e-05 true resid
> norm 2.489381364970e-04 ||r(i)||/||b|| 3.379674620752e-05
> > > > > > 80 KSP preconditioned resid norm 1.309769805765e-05 true resid
> norm 2.489285056062e-04 ||r(i)||/||b|| 3.379543868279e-05
> > > > > > 81 KSP preconditioned resid norm 1.293686496271e-05 true resid
> norm 2.489347818072e-04 ||r(i)||/||b|| 3.379629076264e-05
> > > > > > 82 KSP preconditioned resid norm 1.311788285799e-05 true resid
> norm 2.489320040215e-04 ||r(i)||/||b|| 3.379591364037e-05
> > > > > > 83 KSP preconditioned resid norm 1.313667378798e-05 true resid
> norm 2.489329437217e-04 ||r(i)||/||b|| 3.379604121748e-05
> > > > > > 84 KSP preconditioned resid norm 1.416138205017e-05 true resid
> norm 2.489266908838e-04 ||r(i)||/||b|| 3.379519230948e-05
> > > > > > 85 KSP preconditioned resid norm 1.452253464774e-05 true resid
> norm 2.489285688375e-04 ||r(i)||/||b|| 3.379544726729e-05
> > > > > > 86 KSP preconditioned resid norm 1.426709413370e-05 true resid
> norm 2.489362313402e-04 ||r(i)||/||b|| 3.379648755651e-05
> > > > > > 87 KSP preconditioned resid norm 1.427480849552e-05 true resid
> norm 2.489378183000e-04 ||r(i)||/||b|| 3.379670300795e-05
> > > > > > 88 KSP preconditioned resid norm 1.413870980147e-05 true resid
> norm 2.489325756118e-04 ||r(i)||/||b|| 3.379599124153e-05
> > > > > > 89 KSP preconditioned resid norm 1.353259857657e-05 true resid
> norm 2.489318968308e-04 ||r(i)||/||b|| 3.379589908776e-05
> > > > > > 90 KSP preconditioned resid norm 1.347676448611e-05 true resid
> norm 2.489332074417e-04 ||r(i)||/||b|| 3.379607702106e-05
> > > > > > 91 KSP preconditioned resid norm 1.362825902909e-05 true resid
> norm 2.489344974971e-04 ||r(i)||/||b|| 3.379625216367e-05
> > > > > > 92 KSP preconditioned resid norm 1.346280901052e-05 true resid
> norm 2.489302570131e-04 ||r(i)||/||b|| 3.379567646016e-05
> > > > > > 93 KSP preconditioned resid norm 1.328052169696e-05 true resid
> norm 2.489346601224e-04 ||r(i)||/||b|| 3.379627424228e-05
> > > > > > 94 KSP preconditioned resid norm 1.554682082515e-05 true resid
> norm 2.489309078759e-04 ||r(i)||/||b|| 3.379576482365e-05
> > > > > > 95 KSP preconditioned resid norm 1.557128675775e-05 true resid
> norm 2.489317143582e-04 ||r(i)||/||b|| 3.379587431462e-05
> > > > > > 96 KSP preconditioned resid norm 1.542571813923e-05 true resid
> norm 2.489319910303e-04 ||r(i)||/||b|| 3.379591187663e-05
> > > > > > 97 KSP preconditioned resid norm 1.570516684444e-05 true resid
> norm 2.489321980894e-04 ||r(i)||/||b|| 3.379593998772e-05
> > > > > > 98 KSP preconditioned resid norm 1.600431789899e-05 true resid
> norm 2.489297450311e-04 ||r(i)||/||b|| 3.379560695162e-05
> > > > > > 99 KSP preconditioned resid norm 1.587495554658e-05 true resid
> norm 2.489339000570e-04 ||r(i)||/||b|| 3.379617105303e-05
> > > > > > 100 KSP preconditioned resid norm 1.621163002878e-05 true resid
> norm 2.489299953360e-04 ||r(i)||/||b|| 3.379564093392e-05
> > > > > > 101 KSP preconditioned resid norm 1.627060872574e-05 true resid
> norm 2.489301570161e-04 ||r(i)||/||b|| 3.379566288419e-05
> > > > > > 102 KSP preconditioned resid norm 1.622931647243e-05 true resid
> norm 2.489277930910e-04 ||r(i)||/||b|| 3.379534194913e-05
> > > > > > 103 KSP preconditioned resid norm 1.612544300282e-05 true resid
> norm 2.489317483299e-04 ||r(i)||/||b|| 3.379587892674e-05
> > > > > > 104 KSP preconditioned resid norm 1.880131646630e-05 true resid
> norm 2.489335862583e-04 ||r(i)||/||b|| 3.379612845059e-05
> > > > > > 105 KSP preconditioned resid norm 1.880563295793e-05 true resid
> norm 2.489365017923e-04 ||r(i)||/||b|| 3.379652427408e-05
> > > > > > 106 KSP preconditioned resid norm 1.860619184027e-05 true resid
> norm 2.489362382373e-04 ||r(i)||/||b|| 3.379648849288e-05
> > > > > > 107 KSP preconditioned resid norm 1.877134148719e-05 true resid
> norm 2.489425484523e-04 ||r(i)||/||b|| 3.379734519061e-05
> > > > > > 108 KSP preconditioned resid norm 1.914810713538e-05 true resid
> norm 2.489347415573e-04 ||r(i)||/||b|| 3.379628529818e-05
> > > > > > 109 KSP preconditioned resid norm 1.220673255622e-05 true resid
> norm 2.490196186357e-04 ||r(i)||/||b|| 3.380780851884e-05
> > > > > > 110 KSP preconditioned resid norm 1.215819132910e-05 true resid
> norm 2.490233518072e-04 ||r(i)||/||b|| 3.380831534776e-05
> > > > > > 111 KSP preconditioned resid norm 1.196565427400e-05 true resid
> norm 2.490279438906e-04 ||r(i)||/||b|| 3.380893878570e-05
> > > > > > 112 KSP preconditioned resid norm 1.171748185197e-05 true resid
> norm 2.490242578983e-04 ||r(i)||/||b|| 3.380843836198e-05
> > > > > > 113 KSP preconditioned resid norm 1.162855824118e-05 true resid
> norm 2.490229018536e-04 ||r(i)||/||b|| 3.380825426043e-05
> > > > > > 114 KSP preconditioned resid norm 1.175594685689e-05 true resid
> norm 2.490274328440e-04 ||r(i)||/||b|| 3.380886940415e-05
> > > > > > 115 KSP preconditioned resid norm 1.167979454122e-05 true resid
> norm 2.490271161036e-04 ||r(i)||/||b|| 3.380882640232e-05
> > > > > > 116 KSP preconditioned resid norm 1.181010893019e-05 true resid
> norm 2.490235420657e-04 ||r(i)||/||b|| 3.380834117795e-05
> > > > > > 117 KSP preconditioned resid norm 1.175206638194e-05 true resid
> norm 2.490263165345e-04 ||r(i)||/||b|| 3.380871784992e-05
> > > > > > 118 KSP preconditioned resid norm 1.183804125791e-05 true resid
> norm 2.490221353083e-04 ||r(i)||/||b|| 3.380815019145e-05
> > > > > > 119 KSP preconditioned resid norm 1.186426973727e-05 true resid
> norm 2.490227115336e-04 ||r(i)||/||b|| 3.380822842189e-05
> > > > > > 120 KSP preconditioned resid norm 1.181986776689e-05 true resid
> norm 2.490257884230e-04 ||r(i)||/||b|| 3.380864615159e-05
> > > > > > 121 KSP preconditioned resid norm 1.131443277370e-05 true resid
> norm 2.490259110230e-04 ||r(i)||/||b|| 3.380866279620e-05
> > > > > > 122 KSP preconditioned resid norm 1.114920075859e-05 true resid
> norm 2.490249829382e-04 ||r(i)||/||b|| 3.380853679603e-05
> > > > > > 123 KSP preconditioned resid norm 1.082073321672e-05 true resid
> norm 2.490314084868e-04 ||r(i)||/||b|| 3.380940915187e-05
> > > > > > 124 KSP preconditioned resid norm 3.307785860990e-06 true resid
> norm 2.490613501549e-04 ||r(i)||/||b|| 3.381347414155e-05
> > > > > > 125 KSP preconditioned resid norm 3.287051720572e-06 true resid
> norm 2.490584648195e-04 ||r(i)||/||b|| 3.381308241794e-05
> > > > > > 126 KSP preconditioned resid norm 3.286797046069e-06 true resid
> norm 2.490654396386e-04 ||r(i)||/||b|| 3.381402934473e-05
> > > > > > 127 KSP preconditioned resid norm 3.311592899411e-06 true resid
> norm 2.490627973588e-04 ||r(i)||/||b|| 3.381367061922e-05
> > > > > > 128 KSP preconditioned resid norm 3.560993694635e-06 true resid
> norm 2.490571732816e-04 ||r(i)||/||b|| 3.381290707406e-05
> > > > > > 129 KSP preconditioned resid norm 3.411994617661e-06 true resid
> norm 2.490652122141e-04 ||r(i)||/||b|| 3.381399846875e-05
> > > > > > 130 KSP preconditioned resid norm 3.412383310721e-06 true resid
> norm 2.490633454022e-04 ||r(i)||/||b|| 3.381374502359e-05
> > > > > > 131 KSP preconditioned resid norm 3.288320044878e-06 true resid
> norm 2.490639470096e-04 ||r(i)||/||b|| 3.381382669999e-05
> > > > > > 132 KSP preconditioned resid norm 3.273215756565e-06 true resid
> norm 2.490640390847e-04 ||r(i)||/||b|| 3.381383920043e-05
> > > > > > 133 KSP preconditioned resid norm 3.236969051459e-06 true resid
> norm 2.490678216102e-04 ||r(i)||/||b|| 3.381435272985e-05
> > > > > > 134 KSP preconditioned resid norm 3.203260913942e-06 true resid
> norm 2.490640965346e-04 ||r(i)||/||b|| 3.381384700005e-05
> > > > > > 135 KSP preconditioned resid norm 3.224117152353e-06 true resid
> norm 2.490655026376e-04 ||r(i)||/||b|| 3.381403789770e-05
> > > > > > 136 KSP preconditioned resid norm 3.221577997984e-06 true resid
> norm 2.490684737611e-04 ||r(i)||/||b|| 3.381444126823e-05
> > > > > > 137 KSP preconditioned resid norm 3.195936222128e-06 true resid
> norm 2.490673982333e-04 ||r(i)||/||b|| 3.381429525066e-05
> > > > > > 138 KSP preconditioned resid norm 3.207528137426e-06 true resid
> norm 2.490641247196e-04 ||r(i)||/||b|| 3.381385082655e-05
> > > > > > 139 KSP preconditioned resid norm 3.240134271963e-06 true resid
> norm 2.490615861251e-04 ||r(i)||/||b|| 3.381350617773e-05
> > > > > > 140 KSP preconditioned resid norm 2.698833607230e-06 true resid
> norm 2.490638954889e-04 ||r(i)||/||b|| 3.381381970535e-05
> > > > > > 141 KSP preconditioned resid norm 2.599151209137e-06 true resid
> norm 2.490657106698e-04 ||r(i)||/||b|| 3.381406614091e-05
> > > > > > 142 KSP preconditioned resid norm 2.633939920994e-06 true resid
> norm 2.490707754695e-04 ||r(i)||/||b|| 3.381475375652e-05
> > > > > > 143 KSP preconditioned resid norm 2.519609221376e-06 true resid
> norm 2.490639100480e-04 ||r(i)||/||b|| 3.381382168195e-05
> > > > > > 144 KSP preconditioned resid norm 3.768526937684e-06 true resid
> norm 2.490654096698e-04 ||r(i)||/||b|| 3.381402527606e-05
> > > > > > 145 KSP preconditioned resid norm 3.707841943289e-06 true resid
> norm 2.490630207923e-04 ||r(i)||/||b|| 3.381370095336e-05
> > > > > > 146 KSP preconditioned resid norm 3.698827503486e-06 true resid
> norm 2.490646071561e-04 ||r(i)||/||b|| 3.381391632387e-05
> > > > > > 147 KSP preconditioned resid norm 3.642747039615e-06 true resid
> norm 2.490610990161e-04 ||r(i)||/||b|| 3.381344004604e-05
> > > > > > 148 KSP preconditioned resid norm 3.613100087842e-06 true resid
> norm 2.490617159023e-04 ||r(i)||/||b|| 3.381352379676e-05
> > > > > > 149 KSP preconditioned resid norm 3.637646399299e-06 true resid
> norm 2.490648063023e-04 ||r(i)||/||b|| 3.381394336069e-05
> > > > > > 150 KSP preconditioned resid norm 3.640235367864e-06 true resid
> norm 2.490648516718e-04 ||r(i)||/||b|| 3.381394952022e-05
> > > > > > 151 KSP preconditioned resid norm 3.724708848977e-06 true resid
> norm 2.490622201040e-04 ||r(i)||/||b|| 3.381359224901e-05
> > > > > > 152 KSP preconditioned resid norm 3.665185002770e-06 true resid
> norm 2.490664302790e-04 ||r(i)||/||b|| 3.381416383766e-05
> > > > > > 153 KSP preconditioned resid norm 3.348992579120e-06 true resid
> norm 2.490655722697e-04 ||r(i)||/||b|| 3.381404735121e-05
> > > > > > 154 KSP preconditioned resid norm 3.309431137943e-06 true resid
> norm 2.490727563300e-04 ||r(i)||/||b|| 3.381502268535e-05
> > > > > > 155 KSP preconditioned resid norm 3.299031245428e-06 true resid
> norm 2.490688392843e-04 ||r(i)||/||b|| 3.381449089298e-05
> > > > > > 156 KSP preconditioned resid norm 3.297127463503e-06 true resid
> norm 2.490642207769e-04 ||r(i)||/||b|| 3.381386386763e-05
> > > > > > 157 KSP preconditioned resid norm 3.297370198641e-06 true resid
> norm 2.490666651723e-04 ||r(i)||/||b|| 3.381419572764e-05
> > > > > > 158 KSP preconditioned resid norm 3.290873165210e-06 true resid
> norm 2.490679189538e-04 ||r(i)||/||b|| 3.381436594557e-05
> > > > > > 159 KSP preconditioned resid norm 3.346705292419e-06 true resid
> norm 2.490617329776e-04 ||r(i)||/||b|| 3.381352611496e-05
> > > > > > 160 KSP preconditioned resid norm 3.429583550890e-06 true resid
> norm 2.490675116236e-04 ||r(i)||/||b|| 3.381431064494e-05
> > > > > > 161 KSP preconditioned resid norm 3.425238504679e-06 true resid
> norm 2.490648199058e-04 ||r(i)||/||b|| 3.381394520756e-05
> > > > > > 162 KSP preconditioned resid norm 3.423484857849e-06 true resid
> norm 2.490723208298e-04 ||r(i)||/||b|| 3.381496356025e-05
> > > > > > 163 KSP preconditioned resid norm 3.383655922943e-06 true resid
> norm 2.490659981249e-04 ||r(i)||/||b|| 3.381410516686e-05
> > > > > > 164 KSP preconditioned resid norm 3.477197358452e-06 true resid
> norm 2.490665979073e-04 ||r(i)||/||b|| 3.381418659549e-05
> > > > > > 165 KSP preconditioned resid norm 3.454672202601e-06 true resid
> norm 2.490651358644e-04 ||r(i)||/||b|| 3.381398810323e-05
> > > > > > 166 KSP preconditioned resid norm 3.399075522566e-06 true resid
> norm 2.490678159511e-04 ||r(i)||/||b|| 3.381435196154e-05
> > > > > > 167 KSP preconditioned resid norm 3.305455787400e-06 true resid
> norm 2.490651924523e-04 ||r(i)||/||b|| 3.381399578581e-05
> > > > > > 168 KSP preconditioned resid norm 3.368445533284e-06 true resid
> norm 2.490688061735e-04 ||r(i)||/||b|| 3.381448639774e-05
> > > > > > 169 KSP preconditioned resid norm 2.981519724814e-06 true resid
> norm 2.490676378334e-04 ||r(i)||/||b|| 3.381432777964e-05
> > > > > > 170 KSP preconditioned resid norm 3.034423065539e-06 true resid
> norm 2.490694458885e-04 ||r(i)||/||b|| 3.381457324777e-05
> > > > > > 171 KSP preconditioned resid norm 2.885972780503e-06 true resid
> norm 2.490688033729e-04 ||r(i)||/||b|| 3.381448601752e-05
> > > > > > 172 KSP preconditioned resid norm 2.892491075033e-06 true resid
> norm 2.490692993765e-04 ||r(i)||/||b|| 3.381455335678e-05
> > > > > > 173 KSP preconditioned resid norm 2.921316177611e-06 true resid
> norm 2.490697629787e-04 ||r(i)||/||b|| 3.381461629709e-05
> > > > > > 174 KSP preconditioned resid norm 2.999889222269e-06 true resid
> norm 2.490707272626e-04 ||r(i)||/||b|| 3.381474721178e-05
> > > > > > 175 KSP preconditioned resid norm 2.975590207575e-06 true resid
> norm 2.490685439925e-04 ||r(i)||/||b|| 3.381445080310e-05
> > > > > > 176 KSP preconditioned resid norm 2.983065843597e-06 true resid
> norm 2.490701883671e-04 ||r(i)||/||b|| 3.381467404937e-05
> > > > > > 177 KSP preconditioned resid norm 2.965959610245e-06 true resid
> norm 2.490711538630e-04 ||r(i)||/||b|| 3.381480512861e-05
> > > > > > 178 KSP preconditioned resid norm 3.005389788827e-06 true resid
> norm 2.490751808095e-04 ||r(i)||/||b|| 3.381535184150e-05
> > > > > > 179 KSP preconditioned resid norm 2.956581668772e-06 true resid
> norm 2.490653125636e-04 ||r(i)||/||b|| 3.381401209257e-05
> > > > > > 180 KSP preconditioned resid norm 2.937498883661e-06 true resid
> norm 2.490666056653e-04 ||r(i)||/||b|| 3.381418764874e-05
> > > > > > 181 KSP preconditioned resid norm 2.913227475431e-06 true resid
> norm 2.490682436979e-04 ||r(i)||/||b|| 3.381441003402e-05
> > > > > > 182 KSP preconditioned resid norm 3.048172862254e-06 true resid
> norm 2.490719669872e-04 ||r(i)||/||b|| 3.381491552130e-05
> > > > > > 183 KSP preconditioned resid norm 3.023868104933e-06 true resid
> norm 2.490648745555e-04 ||r(i)||/||b|| 3.381395262699e-05
> > > > > > 184 KSP preconditioned resid norm 2.985947506400e-06 true resid
> norm 2.490638818852e-04 ||r(i)||/||b|| 3.381381785846e-05
> > > > > > 185 KSP preconditioned resid norm 2.840032055776e-06 true resid
> norm 2.490701112392e-04 ||r(i)||/||b|| 3.381466357820e-05
> > > > > > 186 KSP preconditioned resid norm 2.229279683815e-06 true resid
> norm 2.490609220680e-04 ||r(i)||/||b|| 3.381341602292e-05
> > > > > > 187 KSP preconditioned resid norm 2.441513276379e-06 true resid
> norm 2.490674056899e-04 ||r(i)||/||b|| 3.381429626300e-05
> > > > > > 188 KSP preconditioned resid norm 2.467046864016e-06 true resid
> norm 2.490691622632e-04 ||r(i)||/||b|| 3.381453474178e-05
> > > > > > 189 KSP preconditioned resid norm 2.482124586361e-06 true resid
> norm 2.490664992339e-04 ||r(i)||/||b|| 3.381417319923e-05
> > > > > > 190 KSP preconditioned resid norm 2.470564926502e-06 true resid
> norm 2.490617019713e-04 ||r(i)||/||b|| 3.381352190543e-05
> > > > > > 191 KSP preconditioned resid norm 2.457947086578e-06 true resid
> norm 2.490628644250e-04 ||r(i)||/||b|| 3.381367972437e-05
> > > > > > 192 KSP preconditioned resid norm 2.469444741724e-06 true resid
> norm 2.490639416335e-04 ||r(i)||/||b|| 3.381382597011e-05
> > > > > > 193 KSP preconditioned resid norm 2.469951525219e-06 true resid
> norm 2.490599769764e-04 ||r(i)||/||b|| 3.381328771385e-05
> > > > > > 194 KSP preconditioned resid norm 2.467486786643e-06 true resid
> norm 2.490630178622e-04 ||r(i)||/||b|| 3.381370055556e-05
> > > > > > 195 KSP preconditioned resid norm 2.409684391404e-06 true resid
> norm 2.490640302606e-04 ||r(i)||/||b|| 3.381383800245e-05
> > > > > > 196 KSP preconditioned resid norm 2.456046691135e-06 true resid
> norm 2.490637730235e-04 ||r(i)||/||b|| 3.381380307900e-05
> > > > > > 197 KSP preconditioned resid norm 2.300015653805e-06 true resid
> norm 2.490615406913e-04 ||r(i)||/||b|| 3.381350000947e-05
> > > > > > 198 KSP preconditioned resid norm 2.238328275301e-06 true resid
> norm 2.490647641246e-04 ||r(i)||/||b|| 3.381393763449e-05
> > > > > > 199 KSP preconditioned resid norm 2.317293820319e-06 true resid
> norm 2.490641611282e-04 ||r(i)||/||b|| 3.381385576951e-05
> > > > > > 200 KSP preconditioned resid norm 2.359590971314e-06 true resid
> norm 2.490685242974e-04 ||r(i)||/||b|| 3.381444812922e-05
> > > > > > 201 KSP preconditioned resid norm 2.311199691596e-06 true resid
> norm 2.490656791753e-04 ||r(i)||/||b|| 3.381406186510e-05
> > > > > > 202 KSP preconditioned resid norm 2.328772904196e-06 true resid
> norm 2.490651045523e-04 ||r(i)||/||b|| 3.381398385220e-05
> > > > > > 203 KSP preconditioned resid norm 2.332731604717e-06 true resid
> norm 2.490649960574e-04 ||r(i)||/||b|| 3.381396912253e-05
> > > > > > 204 KSP preconditioned resid norm 2.357629383490e-06 true resid
> norm 2.490686317727e-04 ||r(i)||/||b|| 3.381446272046e-05
> > > > > > 205 KSP preconditioned resid norm 2.374856180299e-06 true resid
> norm 2.490645897176e-04 ||r(i)||/||b|| 3.381391395637e-05
> > > > > > 206 KSP preconditioned resid norm 2.340395514404e-06 true resid
> norm 2.490618341127e-04 ||r(i)||/||b|| 3.381353984542e-05
> > > > > > 207 KSP preconditioned resid norm 2.314963680954e-06 true resid
> norm 2.490676153984e-04 ||r(i)||/||b|| 3.381432473379e-05
> > > > > > 208 KSP preconditioned resid norm 2.448070953106e-06 true resid
> norm 2.490644606776e-04 ||r(i)||/||b|| 3.381389643743e-05
> > > > > > 209 KSP preconditioned resid norm 2.428805110632e-06 true resid
> norm 2.490635817597e-04 ||r(i)||/||b|| 3.381377711234e-05
> > > > > > 210 KSP preconditioned resid norm 2.537929937808e-06 true resid
> norm 2.490680589404e-04 ||r(i)||/||b|| 3.381438495066e-05
> > > > > > 211 KSP preconditioned resid norm 2.515909029682e-06 true resid
> norm 2.490687803038e-04 ||r(i)||/||b|| 3.381448288557e-05
> > > > > > 212 KSP preconditioned resid norm 2.497907513266e-06 true resid
> norm 2.490618016885e-04 ||r(i)||/||b|| 3.381353544340e-05
> > > > > > 213 KSP preconditioned resid norm 1.783501869502e-06 true resid
> norm 2.490632647470e-04 ||r(i)||/||b|| 3.381373407354e-05
> > > > > > 214 KSP preconditioned resid norm 1.767420653144e-06 true resid
> norm 2.490685569328e-04 ||r(i)||/||b|| 3.381445255992e-05
> > > > > > 215 KSP preconditioned resid norm 1.854926068272e-06 true resid
> norm 2.490609365464e-04 ||r(i)||/||b|| 3.381341798856e-05
> > > > > > 216 KSP preconditioned resid norm 1.818308539774e-06 true resid
> norm 2.490639142283e-04 ||r(i)||/||b|| 3.381382224948e-05
> > > > > > 217 KSP preconditioned resid norm 1.809431578070e-06 true resid
> norm 2.490605125049e-04 ||r(i)||/||b|| 3.381336041915e-05
> > > > > > 218 KSP preconditioned resid norm 1.789862735999e-06 true resid
> norm 2.490564024901e-04 ||r(i)||/||b|| 3.381280242859e-05
> > > > > > 219 KSP preconditioned resid norm 1.769239890163e-06 true resid
> norm 2.490647825316e-04 ||r(i)||/||b|| 3.381394013349e-05
> > > > > > 220 KSP preconditioned resid norm 1.780760773109e-06 true resid
> norm 2.490622606663e-04 ||r(i)||/||b|| 3.381359775589e-05
> > > > > > 221 KSP preconditioned resid norm 5.009024913368e-07 true resid
> norm 2.490659101637e-04 ||r(i)||/||b|| 3.381409322492e-05
> > > > > > 222 KSP preconditioned resid norm 4.974450322799e-07 true resid
> norm 2.490714287402e-04 ||r(i)||/||b|| 3.381484244693e-05
> > > > > > 223 KSP preconditioned resid norm 4.938819481519e-07 true resid
> norm 2.490665661715e-04 ||r(i)||/||b|| 3.381418228693e-05
> > > > > > 224 KSP preconditioned resid norm 4.973231831266e-07 true resid
> norm 2.490725000995e-04 ||r(i)||/||b|| 3.381498789855e-05
> > > > > > 225 KSP preconditioned resid norm 5.086864036771e-07 true resid
> norm 2.490664132954e-04 ||r(i)||/||b|| 3.381416153192e-05
> > > > > > 226 KSP preconditioned resid norm 5.046954570561e-07 true resid
> norm 2.490698772594e-04 ||r(i)||/||b|| 3.381463181226e-05
> > > > > > 227 KSP preconditioned resid norm 5.086852920874e-07 true resid
> norm 2.490703544723e-04 ||r(i)||/||b|| 3.381469660041e-05
> > > > > > 228 KSP preconditioned resid norm 5.182381756169e-07 true resid
> norm 2.490665200032e-04 ||r(i)||/||b|| 3.381417601896e-05
> > > > > > 229 KSP preconditioned resid norm 5.261455182896e-07 true resid
> norm 2.490697169472e-04 ||r(i)||/||b|| 3.381461004770e-05
> > > > > > 230 KSP preconditioned resid norm 5.265262522400e-07 true resid
> norm 2.490726890541e-04 ||r(i)||/||b|| 3.381501355172e-05
> > > > > > 231 KSP preconditioned resid norm 5.220652263946e-07 true resid
> norm 2.490689325236e-04 ||r(i)||/||b|| 3.381450355149e-05
> > > > > > 232 KSP preconditioned resid norm 5.256466259888e-07 true resid
> norm 2.490694033989e-04 ||r(i)||/||b|| 3.381456747923e-05
> > > > > > 233 KSP preconditioned resid norm 5.443022648374e-07 true resid
> norm 2.490650183144e-04 ||r(i)||/||b|| 3.381397214423e-05
> > > > > > 234 KSP preconditioned resid norm 5.562619006436e-07 true resid
> norm 2.490764576883e-04 ||r(i)||/||b|| 3.381552519520e-05
> > > > > > 235 KSP preconditioned resid norm 5.998148629545e-07 true resid
> norm 2.490714032716e-04 ||r(i)||/||b|| 3.381483898922e-05
> > > > > > 236 KSP preconditioned resid norm 6.498977322955e-07 true resid
> norm 2.490650270144e-04 ||r(i)||/||b|| 3.381397332537e-05
> > > > > > 237 KSP preconditioned resid norm 6.503686003429e-07 true resid
> norm 2.490706976108e-04 ||r(i)||/||b|| 3.381474318615e-05
> > > > > > 238 KSP preconditioned resid norm 6.566719023119e-07 true resid
> norm 2.490664107559e-04 ||r(i)||/||b|| 3.381416118714e-05
> > > > > > 239 KSP preconditioned resid norm 6.549737473208e-07 true resid
> norm 2.490721547909e-04 ||r(i)||/||b|| 3.381494101821e-05
> > > > > > 240 KSP preconditioned resid norm 6.616898981418e-07 true resid
> norm 2.490679659838e-04 ||r(i)||/||b|| 3.381437233053e-05
> > > > > > 241 KSP preconditioned resid norm 6.829917691021e-07 true resid
> norm 2.490728328614e-04 ||r(i)||/||b|| 3.381503307553e-05
> > > > > > 242 KSP preconditioned resid norm 7.030239869389e-07 true resid
> norm 2.490706345115e-04 ||r(i)||/||b|| 3.381473461955e-05
> > > > > > 243 KSP preconditioned resid norm 7.018435683340e-07 true resid
> norm 2.490650978460e-04 ||r(i)||/||b|| 3.381398294172e-05
> > > > > > 244 KSP preconditioned resid norm 7.058047080376e-07 true resid
> norm 2.490685975642e-04 ||r(i)||/||b|| 3.381445807618e-05
> > > > > > 245 KSP preconditioned resid norm 6.896300385099e-07 true resid
> norm 2.490708566380e-04 ||r(i)||/||b|| 3.381476477625e-05
> > > > > > 246 KSP preconditioned resid norm 7.093960074437e-07 true resid
> norm 2.490667427871e-04 ||r(i)||/||b|| 3.381420626490e-05
> > > > > > 247 KSP preconditioned resid norm 7.817121711853e-07 true resid
> norm 2.490692299030e-04 ||r(i)||/||b|| 3.381454392480e-05
> > > > > > 248 KSP preconditioned resid norm 7.976109778309e-07 true resid
> norm 2.490686360729e-04 ||r(i)||/||b|| 3.381446330426e-05
> > > > > > 249 KSP preconditioned resid norm 7.855322750445e-07 true resid
> norm 2.490720861966e-04 ||r(i)||/||b|| 3.381493170560e-05
> > > > > > 250 KSP preconditioned resid norm 7.778531114042e-07 true resid
> norm 2.490673235034e-04 ||r(i)||/||b|| 3.381428510506e-05
> > > > > > 251 KSP preconditioned resid norm 7.848682182070e-07 true resid
> norm 2.490686360729e-04 ||r(i)||/||b|| 3.381446330426e-05
> > > > > > 252 KSP preconditioned resid norm 7.967291867330e-07 true resid
> norm 2.490724820229e-04 ||r(i)||/||b|| 3.381498544442e-05
> > > > > > 253 KSP preconditioned resid norm 7.865012959525e-07 true resid
> norm 2.490666662028e-04 ||r(i)||/||b|| 3.381419586754e-05
> > > > > > 254 KSP preconditioned resid norm 7.656025385804e-07 true resid
> norm 2.490686283214e-04 ||r(i)||/||b|| 3.381446225190e-05
> > > > > > 255 KSP preconditioned resid norm 7.757018653468e-07 true resid
> norm 2.490655983763e-04 ||r(i)||/||b|| 3.381405089553e-05
> > > > > > 256 KSP preconditioned resid norm 6.686490372981e-07 true resid
> norm 2.490715698964e-04 ||r(i)||/||b|| 3.381486161081e-05
> > > > > > 257 KSP preconditioned resid norm 6.596005109428e-07 true resid
> norm 2.490666403003e-04 ||r(i)||/||b|| 3.381419235092e-05
> > > > > > 258 KSP preconditioned resid norm 6.681742296333e-07 true resid
> norm 2.490683725835e-04 ||r(i)||/||b|| 3.381442753198e-05
> > > > > > 259 KSP preconditioned resid norm 1.089245482033e-06 true resid
> norm 2.490688086568e-04 ||r(i)||/||b|| 3.381448673488e-05
> > > > > > 260 KSP preconditioned resid norm 1.099844873189e-06 true resid
> norm 2.490690703265e-04 ||r(i)||/||b|| 3.381452226011e-05
> > > > > > 261 KSP preconditioned resid norm 1.112925540869e-06 true resid
> norm 2.490664481058e-04 ||r(i)||/||b|| 3.381416625790e-05
> > > > > > 262 KSP preconditioned resid norm 1.113056910480e-06 true resid
> norm 2.490658753273e-04 ||r(i)||/||b|| 3.381408849541e-05
> > > > > > 263 KSP preconditioned resid norm 1.104801535149e-06 true resid
> norm 2.490736510776e-04 ||r(i)||/||b|| 3.381514415953e-05
> > > > > > 264 KSP preconditioned resid norm 1.158709147873e-06 true resid
> norm 2.490607531152e-04 ||r(i)||/||b|| 3.381339308528e-05
> > > > > > 265 KSP preconditioned resid norm 1.178985740182e-06 true resid
> norm 2.490727895619e-04 ||r(i)||/||b|| 3.381502719703e-05
> > > > > > 266 KSP preconditioned resid norm 1.165130533478e-06 true resid
> norm 2.490639076693e-04 ||r(i)||/||b|| 3.381382135901e-05
> > > > > > 267 KSP preconditioned resid norm 1.181364114499e-06 true resid
> norm 2.490667871436e-04 ||r(i)||/||b|| 3.381421228690e-05
> > > > > > 268 KSP preconditioned resid norm 1.170295348543e-06 true resid
> norm 2.490662613306e-04 ||r(i)||/||b|| 3.381414090063e-05
> > > > > > 269 KSP preconditioned resid norm 1.213243016230e-06 true resid
> norm 2.490666173719e-04 ||r(i)||/||b|| 3.381418923808e-05
> > > > > > 270 KSP preconditioned resid norm 1.239691953997e-06 true resid
> norm 2.490678323197e-04 ||r(i)||/||b|| 3.381435418381e-05
> > > > > > 271 KSP preconditioned resid norm 1.219891740100e-06 true resid
> norm 2.490625009256e-04 ||r(i)||/||b|| 3.381363037437e-05
> > > > > > 272 KSP preconditioned resid norm 1.231321334346e-06 true resid
> norm 2.490659733696e-04 ||r(i)||/||b|| 3.381410180599e-05
> > > > > > 273 KSP preconditioned resid norm 1.208183234158e-06 true resid
> norm 2.490685987255e-04 ||r(i)||/||b|| 3.381445823385e-05
> > > > > > 274 KSP preconditioned resid norm 1.211768545589e-06 true resid
> norm 2.490671548953e-04 ||r(i)||/||b|| 3.381426221421e-05
> > > > > > 275 KSP preconditioned resid norm 1.209433459842e-06 true resid
> norm 2.490669016096e-04 ||r(i)||/||b|| 3.381422782722e-05
> > > > > > 276 KSP preconditioned resid norm 1.223729184405e-06 true resid
> norm 2.490658128014e-04 ||r(i)||/||b|| 3.381408000666e-05
> > > > > > 277 KSP preconditioned resid norm 1.243915201868e-06 true resid
> norm 2.490693375756e-04 ||r(i)||/||b|| 3.381455854282e-05
> > > > > > 278 KSP preconditioned resid norm 1.231994655529e-06 true resid
> norm 2.490682988311e-04 ||r(i)||/||b|| 3.381441751910e-05
> > > > > > 279 KSP preconditioned resid norm 1.227930683777e-06 true resid
> norm 2.490667825866e-04 ||r(i)||/||b|| 3.381421166823e-05
> > > > > > 280 KSP preconditioned resid norm 1.193458846469e-06 true resid
> norm 2.490687366117e-04 ||r(i)||/||b|| 3.381447695378e-05
> > > > > > 281 KSP preconditioned resid norm 1.217089059805e-06 true resid
> norm 2.490674797371e-04 ||r(i)||/||b|| 3.381430631591e-05
> > > > > > 282 KSP preconditioned resid norm 1.249318287709e-06 true resid
> norm 2.490662866951e-04 ||r(i)||/||b|| 3.381414434420e-05
> > > > > > 283 KSP preconditioned resid norm 1.183320029547e-06 true resid
> norm 2.490645783630e-04 ||r(i)||/||b|| 3.381391241482e-05
> > > > > > 284 KSP preconditioned resid norm 1.174730603102e-06 true resid
> norm 2.490686881647e-04 ||r(i)||/||b|| 3.381447037643e-05
> > > > > > 285 KSP preconditioned resid norm 1.175838261923e-06 true resid
> norm 2.490665969300e-04 ||r(i)||/||b|| 3.381418646281e-05
> > > > > > 286 KSP preconditioned resid norm 1.188946188368e-06 true resid
> norm 2.490661974622e-04 ||r(i)||/||b|| 3.381413222961e-05
> > > > > > 287 KSP preconditioned resid norm 1.177848565707e-06 true resid
> norm 2.490660236206e-04 ||r(i)||/||b|| 3.381410862824e-05
> > > > > > 288 KSP preconditioned resid norm 1.200075508281e-06 true resid
> norm 2.490645353536e-04 ||r(i)||/||b|| 3.381390657571e-05
> > > > > > 289 KSP preconditioned resid norm 1.184589570618e-06 true resid
> norm 2.490664920355e-04 ||r(i)||/||b|| 3.381417222195e-05
> > > > > > 290 KSP preconditioned resid norm 1.221114703873e-06 true resid
> norm 2.490670597538e-04 ||r(i)||/||b|| 3.381424929746e-05
> > > > > > 291 KSP preconditioned resid norm 1.249479658256e-06 true resid
> norm 2.490641582876e-04 ||r(i)||/||b|| 3.381385538385e-05
> > > > > > 292 KSP preconditioned resid norm 1.245768496850e-06 true resid
> norm 2.490704480588e-04 ||r(i)||/||b|| 3.381470930606e-05
> > > > > > 293 KSP preconditioned resid norm 1.243742607953e-06 true resid
> norm 2.490649690604e-04 ||r(i)||/||b|| 3.381396545733e-05
> > > > > > 294 KSP preconditioned resid norm 1.342758483339e-06 true resid
> norm 2.490676207432e-04 ||r(i)||/||b|| 3.381432545942e-05
> > > > > > 295 KSP preconditioned resid norm 1.353816099600e-06 true resid
> norm 2.490695263153e-04 ||r(i)||/||b|| 3.381458416681e-05
> > > > > > 296 KSP preconditioned resid norm 1.343886763293e-06 true resid
> norm 2.490673674307e-04 ||r(i)||/||b|| 3.381429106879e-05
> > > > > > 297 KSP preconditioned resid norm 1.355511022815e-06 true resid
> norm 2.490686565995e-04 ||r(i)||/||b|| 3.381446609103e-05
> > > > > > 298 KSP preconditioned resid norm 1.347247627243e-06 true resid
> norm 2.490696287707e-04 ||r(i)||/||b|| 3.381459807652e-05
> > > > > > 299 KSP preconditioned resid norm 1.414742595618e-06 true resid
> norm 2.490749815091e-04 ||r(i)||/||b|| 3.381532478374e-05
> > > > > > 300 KSP preconditioned resid norm 1.418560683189e-06 true resid
> norm 2.490721501153e-04 ||r(i)||/||b|| 3.381494038343e-05
> > > > > > 301 KSP preconditioned resid norm 1.416276404923e-06 true resid
> norm 2.490689576447e-04 ||r(i)||/||b|| 3.381450696203e-05
> > > > > > 302 KSP preconditioned resid norm 1.431448272112e-06 true resid
> norm 2.490688812701e-04 ||r(i)||/||b|| 3.381449659312e-05
> > > > > > 303 KSP preconditioned resid norm 1.446154958969e-06 true resid
> norm 2.490727536322e-04 ||r(i)||/||b|| 3.381502231909e-05
> > > > > > 304 KSP preconditioned resid norm 1.468860617921e-06 true resid
> norm 2.490692363788e-04 ||r(i)||/||b|| 3.381454480397e-05
> > > > > > 305 KSP preconditioned resid norm 1.627595214971e-06 true resid
> norm 2.490687019603e-04 ||r(i)||/||b|| 3.381447224938e-05
> > > > > > 306 KSP preconditioned resid norm 1.614384672893e-06 true resid
> norm 2.490687019603e-04 ||r(i)||/||b|| 3.381447224938e-05
> > > > > > 307 KSP preconditioned resid norm 1.605568020532e-06 true resid
> norm 2.490699757693e-04 ||r(i)||/||b|| 3.381464518632e-05
> > > > > > 308 KSP preconditioned resid norm 1.617069685075e-06 true resid
> norm 2.490649282923e-04 ||r(i)||/||b|| 3.381395992249e-05
> > > > > > 309 KSP preconditioned resid norm 1.654297792738e-06 true resid
> norm 2.490644766626e-04 ||r(i)||/||b|| 3.381389860760e-05
> > > > > > 310 KSP preconditioned resid norm 1.587528143215e-06 true resid
> norm 2.490696752096e-04 ||r(i)||/||b|| 3.381460438124e-05
> > > > > > 311 KSP preconditioned resid norm 1.662782022388e-06 true resid
> norm 2.490699317737e-04 ||r(i)||/||b|| 3.381463921332e-05
> > > > > > 312 KSP preconditioned resid norm 1.618211471748e-06 true resid
> norm 2.490735831308e-04 ||r(i)||/||b|| 3.381513493483e-05
> > > > > > 313 KSP preconditioned resid norm 1.609074961921e-06 true resid
> norm 2.490679566436e-04 ||r(i)||/||b|| 3.381437106247e-05
> > > > > > 314 KSP preconditioned resid norm 1.548068942878e-06 true resid
> norm 2.490660071226e-04 ||r(i)||/||b|| 3.381410638842e-05
> > > > > > 315 KSP preconditioned resid norm 1.526718322150e-06 true resid
> norm 2.490619832967e-04 ||r(i)||/||b|| 3.381356009919e-05
> > > > > > 316 KSP preconditioned resid norm 1.553150959105e-06 true resid
> norm 2.490660071226e-04 ||r(i)||/||b|| 3.381410638842e-05
> > > > > > 317 KSP preconditioned resid norm 1.615015320906e-06 true resid
> norm 2.490672348079e-04 ||r(i)||/||b|| 3.381427306343e-05
> > > > > > 318 KSP preconditioned resid norm 1.602904469797e-06 true resid
> norm 2.490696731006e-04 ||r(i)||/||b|| 3.381460409491e-05
> > > > > > 319 KSP preconditioned resid norm 1.538140323073e-06 true resid
> norm 2.490722982494e-04 ||r(i)||/||b|| 3.381496049466e-05
> > > > > > 320 KSP preconditioned resid norm 1.534779679430e-06 true resid
> norm 2.490778499789e-04 ||r(i)||/||b|| 3.381571421763e-05
> > > > > > 321 KSP preconditioned resid norm 1.547155843355e-06 true resid
> norm 2.490767612985e-04 ||r(i)||/||b|| 3.381556641442e-05
> > > > > > 322 KSP preconditioned resid norm 1.422137008870e-06 true resid
> norm 2.490737676309e-04 ||r(i)||/||b|| 3.381515998323e-05
> > > > > > 323 KSP preconditioned resid norm 1.403072558954e-06 true resid
> norm 2.490741361870e-04 ||r(i)||/||b|| 3.381521001975e-05
> > > > > > 324 KSP preconditioned resid norm 1.373070436118e-06 true resid
> norm 2.490742214990e-04 ||r(i)||/||b|| 3.381522160202e-05
> > > > > > 325 KSP preconditioned resid norm 1.359547585233e-06 true resid
> norm 2.490792987570e-04 ||r(i)||/||b|| 3.381591090902e-05
> > > > > > 326 KSP preconditioned resid norm 1.370351913612e-06 true resid
> norm 2.490727161158e-04 ||r(i)||/||b|| 3.381501722573e-05
> > > > > > 327 KSP preconditioned resid norm 1.365238666187e-06 true resid
> norm 2.490716949642e-04 ||r(i)||/||b|| 3.381487859046e-05
> > > > > > 328 KSP preconditioned resid norm 1.369073373042e-06 true resid
> norm 2.490807360288e-04 ||r(i)||/||b|| 3.381610603826e-05
> > > > > > 329 KSP preconditioned resid norm 1.426698981572e-06 true resid
> norm 2.490791479521e-04 ||r(i)||/||b|| 3.381589043520e-05
> > > > > > 330 KSP preconditioned resid norm 1.445542403570e-06 true resid
> norm 2.490775981409e-04 ||r(i)||/||b|| 3.381568002720e-05
> > > > > > 331 KSP preconditioned resid norm 1.464506963984e-06 true resid
> norm 2.490740562430e-04 ||r(i)||/||b|| 3.381519916626e-05
> > > > > > 332 KSP preconditioned resid norm 1.461462964401e-06 true resid
> norm 2.490768016856e-04 ||r(i)||/||b|| 3.381557189753e-05
> > > > > > 333 KSP preconditioned resid norm 1.476680847971e-06 true resid
> norm 2.490744321516e-04 ||r(i)||/||b|| 3.381525020097e-05
> > > > > > 334 KSP preconditioned resid norm 1.459640372198e-06 true resid
> norm 2.490788817993e-04 ||r(i)||/||b|| 3.381585430132e-05
> > > > > > 335 KSP preconditioned resid norm 1.790770882365e-06 true resid
> norm 2.490771711471e-04 ||r(i)||/||b|| 3.381562205697e-05
> > > > > > 336 KSP preconditioned resid norm 1.803770155018e-06 true resid
> norm 2.490768953858e-04 ||r(i)||/||b|| 3.381558461860e-05
> > > > > > 337 KSP preconditioned resid norm 1.787821255995e-06 true resid
> norm 2.490767985676e-04 ||r(i)||/||b|| 3.381557147421e-05
> > > > > > 338 KSP preconditioned resid norm 1.749912220831e-06 true resid
> norm 2.490760198704e-04 ||r(i)||/||b|| 3.381546575545e-05
> > > > > > 339 KSP preconditioned resid norm 1.802915839010e-06 true resid
> norm 2.490815556273e-04 ||r(i)||/||b|| 3.381621730993e-05
> > > > > > 340 KSP preconditioned resid norm 1.800777670709e-06 true resid
> norm 2.490823909286e-04 ||r(i)||/||b|| 3.381633071347e-05
> > > > > > 341 KSP preconditioned resid norm 1.962516327690e-06 true resid
> norm 2.490773477410e-04 ||r(i)||/||b|| 3.381564603199e-05
> > > > > > 342 KSP preconditioned resid norm 1.981726465132e-06 true resid
> norm 2.490769116884e-04 ||r(i)||/||b|| 3.381558683191e-05
> > > > > > 343 KSP preconditioned resid norm 1.963419167052e-06 true resid
> norm 2.490764009914e-04 ||r(i)||/||b|| 3.381551749783e-05
> > > > > > 344 KSP preconditioned resid norm 1.992082169278e-06 true resid
> norm 2.490806829883e-04 ||r(i)||/||b|| 3.381609883728e-05
> > > > > > 345 KSP preconditioned resid norm 1.981005134253e-06 true resid
> norm 2.490748677339e-04 ||r(i)||/||b|| 3.381530933721e-05
> > > > > > 346 KSP preconditioned resid norm 1.959802663114e-06 true resid
> norm 2.490773752317e-04 ||r(i)||/||b|| 3.381564976423e-05
> > > > > >
> > > > > > On Sat, Oct 21, 2017 at 5:25 PM, Matthew Knepley <
> knepley at gmail.com> wrote:
> > > > > > On Sat, Oct 21, 2017 at 5:21 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > > > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
> > > > > > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
> > > > > >
> > > > > > ierr = VecAssemblyBegin(x);
> > > > > > ierr = VecAssemblyEnd(x);
> > > > > > This is probably unnecessary
> > > > > >
> > > > > > ierr = VecAssemblyBegin(b);
> > > > > > ierr = VecAssemblyEnd(b);
> > > > > > This is probably unnecessary
> > > > > >
> > > > > >
> > > > > > ierr = MatNullSpaceCreate(PETSC_COMM_
> WORLD,PETSC_TRUE,0,PETSC_NULL,&nullsp);
> > > > > > ierr = MatSetNullSpace(A,nullsp); // Petsc-3.8
> > > > > > Is your rhs consistent with this nullspace?
> > > > > >
> > > > > > // KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);
> > > > > > KSPSetOperators(ksp,A,A);
> > > > > >
> > > > > > KSPSetType(ksp,KSPBCGS);
> > > > > >
> > > > > > KSPSetComputeSingularValues(ksp, PETSC_TRUE);
> > > > > > #if defined(__HYPRE__)
> > > > > > KSPGetPC(ksp, &pc);
> > > > > > PCSetType(pc, PCHYPRE);
> > > > > > PCHYPRESetType(pc,"boomeramg");
> > > > > > This is terribly unnecessary. You just use
> > > > > >
> > > > > > -pc_type hypre -pc_hypre_type boomeramg
> > > > > >
> > > > > > or
> > > > > >
> > > > > > -pc_type gamg
> > > > > >
> > > > > > #else
> > > > > > KSPSetType(ksp,KSPBCGSL);
> > > > > > KSPBCGSLSetEll(ksp,2);
> > > > > > #endif /* defined(__HYPRE__) */
> > > > > >
> > > > > > KSPSetFromOptions(ksp);
> > > > > > KSPSetUp(ksp);
> > > > > >
> > > > > > ierr = KSPSolve(ksp,b,x);
> > > > > >
> > > > > >
> > > > > > command line
> > > > > >
> > > > > > You did not provide any of what I asked for the in the eprevious
> mail.
> > > > > >
> > > > > > Matt
> > > > > >
> > > > > > On Sat, Oct 21, 2017 at 5:16 PM, Matthew Knepley <
> knepley at gmail.com> wrote:
> > > > > > On Sat, Oct 21, 2017 at 5:04 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > > > hi,
> > > > > >
> > > > > > I implemented HYPRE preconditioner for my study due to the fact
> that without preconditioner, PETSc solver will take thousands of iterations
> to converge for fine grid simulation.
> > > > > >
> > > > > > with HYPRE, depending on the parallel partition, it will take
> HYPRE forever to do anything. observation of output file is that the
> simulation is hanging with no output.
> > > > > >
> > > > > > Any idea what happened? will post snippet of code.
> > > > > >
> > > > > > 1) For any question about convergence, we need to see the output
> of
> > > > > >
> > > > > > -ksp_view_pre -ksp_view -ksp_monitor_true_residual
> -ksp_converged_reason
> > > > > >
> > > > > > 2) Hypre has many preconditioners, which one are you talking
> about
> > > > > >
> > > > > > 3) PETSc has some preconditioners in common with Hypre, like AMG
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Matt
> > > > > >
> > > > > > --
> > > > > > Hao Zhang
> > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > Stony Brook University,
> > > > > > Stony Brook, New York, 11790
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > > > > -- Norbert Wiener
> > > > > >
> > > > > > https://www.cse.buffalo.edu/~knepley/
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Hao Zhang
> > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > Stony Brook University,
> > > > > > Stony Brook, New York, 11790
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > > > > -- Norbert Wiener
> > > > > >
> > > > > > https://www.cse.buffalo.edu/~knepley/
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Hao Zhang
> > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > Stony Brook University,
> > > > > > Stony Brook, New York, 11790
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Hao Zhang
> > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > Stony Brook University,
> > > > > > Stony Brook, New York, 11790
> > > > >
> > > > > --
> > > > > Hao Zhang
> > > > > Dept. of Applid Mathematics and Statistics,
> > > > > Stony Brook University,
> > > > > Stony Brook, New York, 11790
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Hao Zhang
> > > > Dept. of Applid Mathematics and Statistics,
> > > > Stony Brook University,
> > > > Stony Brook, New York, 11790
> > >
> > >
> > >
> > >
> > > --
> > > Hao Zhang
> > > Dept. of Applid Mathematics and Statistics,
> > > Stony Brook University,
> > > Stony Brook, New York, 11790
> >
> >
> >
> >
> > --
> > Hao Zhang
> > Dept. of Applid Mathematics and Statistics,
> > Stony Brook University,
> > Stony Brook, New York, 11790
>
>
--
Hao Zhang
Dept. of Applid Mathematics and Statistics,
Stony Brook University,
Stony Brook, New York, 11790
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171022/30530601/attachment-0001.html>
More information about the petsc-users
mailing list