[petsc-users] HYPRE hanging or slow? from observation
Hao Zhang
hbcbh1999 at gmail.com
Sun Oct 22 14:35:45 CDT 2017
Thanks for all the inputs. before simulating finer grid, HYPRE wasn't used
and simulations were just fine. I will do a few tests and post more
information later.
On Sun, Oct 22, 2017 at 12:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> > On Oct 21, 2017, at 11:16 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> >
> > the reason is when I do finer grid simulation, matrix become more stiff.
>
> Are you saying that for a finer grid but everything else the same, the
> convergence of hypre (with the same GMRES) with the same options gets much
> worse? This normally will not happen, that is the fundamental beauty of
> multigrid methods (when they work well).
>
> Yes the matrix condition number increases but multigrid doesn't care
> about that, its number of iterations should remain pretty much the same.
>
> Something must be different (with this finer grid case), either the
> mesh becomes horrible, or the physics changes, or there are errors in the
> code that lead to the problem.
>
> What happens if you just refine the mesh a little? Then a little
> more? Then a little more? Does the convergence rate suddenly go bad at some
> point, or does it just get worse slowly?
>
> Barry
>
>
>
> > Much larger condition number. just to give you a perspective, it will
> take 6000 iterations to converge and the solver does converge. I want to
> reduce the number of iterations while keeping the convergence rate. that's
> main drive to do so much heavy lifting around. please advise. snippet will
> be provided upon request.
> >
> > Thanks again.
> >
> > On Sun, Oct 22, 2017 at 12:08 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >
> > Oh, you change KSP but not hypre. I did not understand this.
> >
> > Why not just use GMRES all the time? Why mess with BCGS if it is not
> robust? Not worth the small optimization if it breaks everything.
> >
> > Barry
> >
> >
> >
> > > On Oct 21, 2017, at 11:05 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > >
> > > this is the initial pressure solver output regarding use of PETSc. it
> failed to converge after 40000 iterations, then use GMRES.
> > >
> > > 39987 KSP preconditioned resid norm 3.853125986269e-08 true resid norm
> 1.147359212216e-05 ||r(i)||/||b|| 1.557696568706e-06
> > > 39988 KSP preconditioned resid norm 3.853126044003e-08 true resid norm
> 1.147359257282e-05 ||r(i)||/||b|| 1.557696629889e-06
> > > 39989 KSP preconditioned resid norm 3.853126052100e-08 true resid norm
> 1.147359233695e-05 ||r(i)||/||b|| 1.557696597866e-06
> > > 39990 KSP preconditioned resid norm 3.853126027357e-08 true resid norm
> 1.147359219860e-05 ||r(i)||/||b|| 1.557696579083e-06
> > > 39991 KSP preconditioned resid norm 3.853126058478e-08 true resid norm
> 1.147359234281e-05 ||r(i)||/||b|| 1.557696598662e-06
> > > 39992 KSP preconditioned resid norm 3.853126064006e-08 true resid norm
> 1.147359261420e-05 ||r(i)||/||b|| 1.557696635506e-06
> > > 39993 KSP preconditioned resid norm 3.853126050203e-08 true resid norm
> 1.147359235972e-05 ||r(i)||/||b|| 1.557696600957e-06
> > > 39994 KSP preconditioned resid norm 3.853126050182e-08 true resid norm
> 1.147359253713e-05 ||r(i)||/||b|| 1.557696625043e-06
> > > 39995 KSP preconditioned resid norm 3.853125976795e-08 true resid norm
> 1.147359226222e-05 ||r(i)||/||b|| 1.557696587720e-06
> > > 39996 KSP preconditioned resid norm 3.853125805127e-08 true resid norm
> 1.147359262747e-05 ||r(i)||/||b|| 1.557696637308e-06
> > > 39997 KSP preconditioned resid norm 3.853125811756e-08 true resid norm
> 1.147359216008e-05 ||r(i)||/||b|| 1.557696573853e-06
> > > 39998 KSP preconditioned resid norm 3.853125827833e-08 true resid norm
> 1.147359238372e-05 ||r(i)||/||b|| 1.557696604216e-06
> > > 39999 KSP preconditioned resid norm 3.853127937068e-08 true resid norm
> 1.147359264043e-05 ||r(i)||/||b|| 1.557696639067e-06
> > > 40000 KSP preconditioned resid norm 3.853122732867e-08 true resid norm
> 1.147359257776e-05 ||r(i)||/||b|| 1.557696630559e-06
> > > Linear solve did not converge due to DIVERGED_ITS iterations 40000
> > > KSP Object: 24 MPI processes
> > > type: bcgs
> > > maximum iterations=40000, initial guess is zero
> > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > left preconditioning
> > > using PRECONDITIONED norm type for convergence test
> > > PC Object: 24 MPI processes
> > > type: hypre
> > > HYPRE BoomerAMG preconditioning
> > > Cycle type V
> > > Maximum number of levels 25
> > > Maximum number of iterations PER hypre call 1
> > > Convergence tolerance PER hypre call 0.
> > > Threshold for strong coupling 0.25
> > > Interpolation truncation factor 0.
> > > Interpolation: max elements per row 0
> > > Number of levels of aggressive coarsening 0
> > > Number of paths for aggressive coarsening 1
> > > Maximum row sums 0.9
> > > Sweeps down 1
> > > Sweeps up 1
> > > Sweeps on coarse 1
> > > Relax down symmetric-SOR/Jacobi
> > > Relax up symmetric-SOR/Jacobi
> > > Relax on coarse Gaussian-elimination
> > > Relax weight (all) 1.
> > > Outer relax weight (all) 1.
> > > Using CF-relaxation
> > > Not using more complex smoothers.
> > > Measure type local
> > > Coarsen type Falgout
> > > Interpolation type classical
> > > linear system matrix = precond matrix:
> > > Mat Object: A 24 MPI processes
> > > type: mpiaij
> > > rows=497664, cols=497664
> > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > total number of mallocs used during MatSetValues calls =0
> > > has attached null space
> > > not using I-node (on process 0) routines
> > >
> > > The solution diverges for p0! The residual is 3.853123e-08. Solve
> again using GMRES!
> > > KSP Object: 24 MPI processes
> > > type: gmres
> > > restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > > happy breakdown tolerance 1e-30
> > > maximum iterations=40000, initial guess is zero
> > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > left preconditioning
> > > using PRECONDITIONED norm type for convergence test
> > > PC Object: 24 MPI processes
> > > type: hypre
> > > HYPRE BoomerAMG preconditioning
> > > Cycle type V
> > > Maximum number of levels 25
> > > Maximum number of iterations PER hypre call 1
> > > Convergence tolerance PER hypre call 0.
> > > Threshold for strong coupling 0.25
> > > Interpolation truncation factor 0.
> > > Interpolation: max elements per row 0
> > > Number of levels of aggressive coarsening 0
> > > Number of paths for aggressive coarsening 1
> > > Maximum row sums 0.9
> > > Sweeps down 1
> > > Sweeps up 1
> > > Sweeps on coarse 1
> > > Relax down symmetric-SOR/Jacobi
> > > Relax up symmetric-SOR/Jacobi
> > > Relax on coarse Gaussian-elimination
> > > Relax weight (all) 1.
> > > Outer relax weight (all) 1.
> > > Using CF-relaxation
> > > Not using more complex smoothers.
> > > Measure type local
> > > Coarsen type Falgout
> > > Interpolation type classical
> > > linear system matrix = precond matrix:
> > > Mat Object: A 24 MPI processes
> > > type: mpiaij
> > > rows=497664, cols=497664
> > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > total number of mallocs used during MatSetValues calls =0
> > > has attached null space
> > > not using I-node (on process 0) routines
> > > 0 KSP preconditioned resid norm 1.593802941804e+01 true resid norm
> 7.365742695119e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > 1 KSP preconditioned resid norm 6.338666661133e-01 true resid norm
> 2.880722209358e+00 ||r(i)||/||b|| 3.910973174867e-01
> > > 2 KSP preconditioned resid norm 3.913420828350e-02 true resid norm
> 9.544089760671e-01 ||r(i)||/||b|| 1.295740315093e-01
> > > 3 KSP preconditioned resid norm 2.928070366435e-03 true resid norm
> 1.874294004628e-01 ||r(i)||/||b|| 2.544609664236e-02
> > > 4 KSP preconditioned resid norm 2.165607525823e-04 true resid norm
> 3.121122463949e-02 ||r(i)||/||b|| 4.237349298146e-03
> > > 5 KSP preconditioned resid norm 1.635476480407e-05 true resid norm
> 3.984315313831e-03 ||r(i)||/||b|| 5.409251284967e-04
> > > 6 KSP preconditioned resid norm 1.283358350575e-06 true resid norm
> 4.566583802915e-04 ||r(i)||/||b|| 6.199760149022e-05
> > > 7 KSP preconditioned resid norm 8.479469225747e-08 true resid norm
> 3.824581791112e-05 ||r(i)||/||b|| 5.192391248810e-06
> > > 8 KSP preconditioned resid norm 5.358636504683e-09 true resid norm
> 2.829730442033e-06 ||r(i)||/||b|| 3.841744898187e-07
> > > 9 KSP preconditioned resid norm 3.447874504193e-10 true resid norm
> 1.756617036538e-07 ||r(i)||/||b|| 2.384847135242e-08
> > > 10 KSP preconditioned resid norm 2.480228743414e-11 true resid norm
> 1.309399823577e-08 ||r(i)||/||b|| 1.777688792258e-09
> > > 11 KSP preconditioned resid norm 1.728967759950e-12 true resid norm
> 9.406967045789e-10 ||r(i)||/||b|| 1.277124036931e-10
> > > 12 KSP preconditioned resid norm 1.075458632828e-13 true resid norm
> 5.994505136212e-11 ||r(i)||/||b|| 8.138358050689e-12
> > > Linear solve converged due to CONVERGED_RTOL iterations 12
> > > KSP Object: 24 MPI processes
> > > type: gmres
> > > restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > > happy breakdown tolerance 1e-30
> > > maximum iterations=40000, initial guess is zero
> > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > left preconditioning
> > > using PRECONDITIONED norm type for convergence test
> > > PC Object: 24 MPI processes
> > > type: hypre
> > > HYPRE BoomerAMG preconditioning
> > > Cycle type V
> > > Maximum number of levels 25
> > > Maximum number of iterations PER hypre call 1
> > > Convergence tolerance PER hypre call 0.
> > > Threshold for strong coupling 0.25
> > > Interpolation truncation factor 0.
> > > Interpolation: max elements per row 0
> > > Number of levels of aggressive coarsening 0
> > > Number of paths for aggressive coarsening 1
> > > Maximum row sums 0.9
> > > Sweeps down 1
> > > Sweeps up 1
> > > Sweeps on coarse 1
> > > Relax down symmetric-SOR/Jacobi
> > > Relax up symmetric-SOR/Jacobi
> > > Relax on coarse Gaussian-elimination
> > > Relax weight (all) 1.
> > > Outer relax weight (all) 1.
> > > Using CF-relaxation
> > > Not using more complex smoothers.
> > > Measure type local
> > > Coarsen type Falgout
> > > Interpolation type classical
> > > linear system matrix = precond matrix:
> > > Mat Object: A 24 MPI processes
> > > type: mpiaij
> > > rows=497664, cols=497664
> > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > total number of mallocs used during MatSetValues calls =0
> > > has attached null space
> > > not using I-node (on process 0) routines
> > > The max singular value of A = 1.000872 in poisson_solver3d_P0_vd
> > > The min singular value of A = 0.667688 in poisson_solver3d_P0_vd
> > > The Cond Num of A = 1.499012 in poisson_solver3d_P0_vd
> > > In poisson_solver3d_pressure(): num_iter = 12, rel_residual =
> 1.075459e-13
> > >
> > > The max value of p0 is 0.03115845493408858
> > >
> > > The min value of p0 is -0.07156715468428149
> > >
> > > On Sun, Oct 22, 2017 at 12:00 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > > > On Oct 21, 2017, at 10:50 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > >
> > > > the incompressible NS solver algorithm call PETSc solver at
> different stage of each time step. The one you were saying "This is good.
> 12 digit reduction" is after the initial pressure solver, in which usually
> HYPRE doesn't give a good convergence, so the fall-back solver GMRES will
> be called after.
> > >
> > > Hmm, I don't understand. hypre should do well on a pressure solve.
> In fact, very well.
> > > >
> > > > Barry, you were mentioning that I could have a wrong nullspace. that
> particular solver is aimed to give an initial pressure profile for 3d
> incompressible NS simulation using all neumann boundary conditions. could
> you give some insight how to test if I have a wrong nullspace etc?
> > >
> > > -ksp_test_null_space
> > >
> > > But if your null space is consistently from all Neumann boundary
> conditions then it likely is not wrong.
> > >
> > > Barry
> > >
> > > >
> > > > Thanks!
> > > >
> > > > On Sat, Oct 21, 2017 at 11:41 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > >
> > > > This is good. You get more than 12 digit reduction in the true
> residual norm. This is good AMG convergence. Expected when everything goes
> well.
> > > >
> > > > What is different in this case from the previous case that does
> not converge reasonably?
> > > >
> > > > Barry
> > > >
> > > >
> > > > > On Oct 21, 2017, at 9:29 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > >
> > > > > Barry, Please advise what you make of this? this is poisson
> solver with all neumann BC 3d case Finite difference Scheme was used.
> > > > > Thanks! I'm in learning mode.
> > > > >
> > > > > KSP Object: 24 MPI processes
> > > > > type: bcgs
> > > > > maximum iterations=40000, initial guess is zero
> > > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > > left preconditioning
> > > > > using PRECONDITIONED norm type for convergence test
> > > > > PC Object: 24 MPI processes
> > > > > type: hypre
> > > > > HYPRE BoomerAMG preconditioning
> > > > > Cycle type V
> > > > > Maximum number of levels 25
> > > > > Maximum number of iterations PER hypre call 1
> > > > > Convergence tolerance PER hypre call 0.
> > > > > Threshold for strong coupling 0.25
> > > > > Interpolation truncation factor 0.
> > > > > Interpolation: max elements per row 0
> > > > > Number of levels of aggressive coarsening 0
> > > > > Number of paths for aggressive coarsening 1
> > > > > Maximum row sums 0.9
> > > > > Sweeps down 1
> > > > > Sweeps up 1
> > > > > Sweeps on coarse 1
> > > > > Relax down symmetric-SOR/Jacobi
> > > > > Relax up symmetric-SOR/Jacobi
> > > > > Relax on coarse Gaussian-elimination
> > > > > Relax weight (all) 1.
> > > > > Outer relax weight (all) 1.
> > > > > Using CF-relaxation
> > > > > Not using more complex smoothers.
> > > > > Measure type local
> > > > > Coarsen type Falgout
> > > > > Interpolation type classical
> > > > > linear system matrix = precond matrix:
> > > > > Mat Object: A 24 MPI processes
> > > > > type: mpiaij
> > > > > rows=497664, cols=497664
> > > > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > > > total number of mallocs used during MatSetValues calls =0
> > > > > has attached null space
> > > > > not using I-node (on process 0) routines
> > > > > 0 KSP preconditioned resid norm 2.697270170623e-02 true resid
> norm 9.637159071207e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > > > 1 KSP preconditioned resid norm 4.828857674609e-04 true resid
> norm 6.294379664645e-01 ||r(i)||/||b|| 6.531364293291e-02
> > > > > 2 KSP preconditioned resid norm 4.533649595815e-06 true resid
> norm 1.135508605857e-02 ||r(i)||/||b|| 1.178260727531e-03
> > > > > 3 KSP preconditioned resid norm 1.131704082606e-07 true resid
> norm 1.196393029874e-04 ||r(i)||/||b|| 1.241437462051e-05
> > > > > 4 KSP preconditioned resid norm 3.866281379569e-10 true resid
> norm 5.121520801846e-07 ||r(i)||/||b|| 5.314347064320e-08
> > > > > 5 KSP preconditioned resid norm 1.070114785241e-11 true resid
> norm 1.203693733135e-08 ||r(i)||/||b|| 1.249013038221e-09
> > > > > 6 KSP preconditioned resid norm 2.578780418765e-14 true resid
> norm 6.020297525927e-11 ||r(i)||/||b|| 6.246962908306e-12
> > > > > 7 KSP preconditioned resid norm 8.691764190203e-16 true resid
> norm 1.866088098154e-12 ||r(i)||/||b|| 1.936346680973e-13
> > > > > Linear solve converged due to CONVERGED_ATOL iterations 7
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Oct 21, 2017 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > >
> > > > > > On Oct 21, 2017, at 5:47 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > > >
> > > > > > hi, Barry:
> > > > > > what do you mean absurd by setting tolerance =1e-14?
> > > > >
> > > > > Trying to decrease the initial residual norm down by a factor of
> 1e-14 with an iterative method (or even direct method) is unrealistic,
> usually unachievable) and almost never necessary. You are requiring || r_n
> || < 1.e-14 || r_0|| when with double precision numbers you only have
> roughly 14 decimal digits total to compute with. Round off alone will lead
> to differences far larger than 1e-14
> > > > >
> > > > > If you are using the solver in the context of a nonlinear
> problem (i.e. inside Newton's method) then 1.e-6 is generally more than
> plenty to get quadratic convergence of Newton's method.
> > > > >
> > > > > If you are solving a linear problem then it is extremely likely
> that errors due to discretization errors (from finite element method etc)
> and the model are much much larger than even 1.e-8.
> > > > >
> > > > > So, in summary
> > > > >
> > > > > 1.e-14 is probably unachievable
> > > > >
> > > > > 1.e-14 is almost for sure not needed.
> > > > >
> > > > > Barry
> > > > >
> > > > >
> > > > > >
> > > > > > On Sat, Oct 21, 2017 at 18:42 Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > > >
> > > > > > Run with -ksp_view_mat binary -ksp_view_rhs binary and send
> the resulting output file called binaryoutput to petsc-maint at mcs.anl.gov
> > > > > >
> > > > > > Note you can also use -ksp_type gmres with hypre, unlikely to
> be a reason to use bcgs
> > > > > >
> > > > > > BTW: tolerances: relative=1e-14, is absurd
> > > > > >
> > > > > > My guess is your null space is incorrect.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On Oct 21, 2017, at 4:34 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > > > >
> > > > > > > if this solver doesn't converge. I have a fall-back solution,
> which uses GMRES solver. this setup is fine with me. I just want to know if
> HYPRE is a reliable solution for me. Or I will have to go without
> preconditioner.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > On Sat, Oct 21, 2017 at 5:30 PM, Hao Zhang <
> hbcbh1999 at gmail.com> wrote:
> > > > > > > this is serial run. still dumping output. parallel more or
> less the same.
> > > > > > >
> > > > > > > KSP Object: 1 MPI processes
> > > > > > > type: bcgs
> > > > > > > maximum iterations=40000, initial guess is zero
> > > > > > > tolerances: relative=1e-14, absolute=1e-14,
> divergence=10000.
> > > > > > > left preconditioning
> > > > > > > using PRECONDITIONED norm type for convergence test
> > > > > > > PC Object: 1 MPI processes
> > > > > > > type: hypre
> > > > > > > HYPRE BoomerAMG preconditioning
> > > > > > > Cycle type V
> > > > > > > Maximum number of levels 25
> > > > > > > Maximum number of iterations PER hypre call 1
> > > > > > > Convergence tolerance PER hypre call 0.
> > > > > > > Threshold for strong coupling 0.25
> > > > > > > Interpolation truncation factor 0.
> > > > > > > Interpolation: max elements per row 0
> > > > > > > Number of levels of aggressive coarsening 0
> > > > > > > Number of paths for aggressive coarsening 1
> > > > > > > Maximum row sums 0.9
> > > > > > > Sweeps down 1
> > > > > > > Sweeps up 1
> > > > > > > Sweeps on coarse 1
> > > > > > > Relax down symmetric-SOR/Jacobi
> > > > > > > Relax up symmetric-SOR/Jacobi
> > > > > > > Relax on coarse Gaussian-elimination
> > > > > > > Relax weight (all) 1.
> > > > > > > Outer relax weight (all) 1.
> > > > > > > Using CF-relaxation
> > > > > > > Not using more complex smoothers.
> > > > > > > Measure type local
> > > > > > > Coarsen type Falgout
> > > > > > > Interpolation type classical
> > > > > > > linear system matrix = precond matrix:
> > > > > > > Mat Object: A 1 MPI processes
> > > > > > > type: seqaij
> > > > > > > rows=497664, cols=497664
> > > > > > > total: nonzeros=3363552, allocated nonzeros=3483648
> > > > > > > total number of mallocs used during MatSetValues calls =0
> > > > > > > has attached null space
> > > > > > > not using I-node routines
> > > > > > > 0 KSP preconditioned resid norm 1.630377897956e+01 true
> resid norm 7.365742695123e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > > > > > 1 KSP preconditioned resid norm 3.815819909205e-02 true
> resid norm 7.592300567353e-01 ||r(i)||/||b|| 1.030758320187e-01
> > > > > > > 2 KSP preconditioned resid norm 4.312671277701e-04 true
> resid norm 2.553060965521e-02 ||r(i)||/||b|| 3.466128360975e-03
> > > > > > > 3 KSP preconditioned resid norm 3.011875330569e-04 true
> resid norm 6.836208627386e-04 ||r(i)||/||b|| 9.281085303065e-05
> > > > > > > 4 KSP preconditioned resid norm 3.011783821295e-04 true
> resid norm 2.504661140204e-04 ||r(i)||/||b|| 3.400418998972e-05
> > > > > > > 5 KSP preconditioned resid norm 3.011783818372e-04 true
> resid norm 2.504053673004e-04 ||r(i)||/||b|| 3.399594279422e-05
> > > > > > > 6 KSP preconditioned resid norm 3.011783818375e-04 true
> resid norm 2.503984078890e-04 ||r(i)||/||b|| 3.399499795925e-05
> > > > > > > 7 KSP preconditioned resid norm 3.011783887442e-04 true
> resid norm 2.504121501772e-04 ||r(i)||/||b|| 3.399686366224e-05
> > > > > > > 8 KSP preconditioned resid norm 3.010913654181e-04 true
> resid norm 2.504150259031e-04 ||r(i)||/||b|| 3.399725408124e-05
> > > > > > > 9 KSP preconditioned resid norm 3.006520688232e-04 true
> resid norm 2.504061607382e-04 ||r(i)||/||b|| 3.399605051423e-05
> > > > > > > 10 KSP preconditioned resid norm 3.007309991942e-04 true
> resid norm 2.503843638523e-04 ||r(i)||/||b|| 3.399309128978e-05
> > > > > > > 11 KSP preconditioned resid norm 3.015946168077e-04 true
> resid norm 2.503644677844e-04 ||r(i)||/||b|| 3.399039012728e-05
> > > > > > > 12 KSP preconditioned resid norm 2.956643907377e-04 true
> resid norm 2.503863046509e-04 ||r(i)||/||b|| 3.399335477965e-05
> > > > > > > 13 KSP preconditioned resid norm 2.997992358936e-04 true
> resid norm 2.504336903093e-04 ||r(i)||/||b|| 3.399978802886e-05
> > > > > > > 14 KSP preconditioned resid norm 2.481415420420e-05 true
> resid norm 2.491591201250e-04 ||r(i)||/||b|| 3.382674774806e-05
> > > > > > > 15 KSP preconditioned resid norm 2.615494786181e-05 true
> resid norm 2.490353237273e-04 ||r(i)||/||b|| 3.380994069915e-05
> > > > > > > 16 KSP preconditioned resid norm 2.645126692130e-05 true
> resid norm 2.490535523344e-04 ||r(i)||/||b|| 3.381241548111e-05
> > > > > > > 17 KSP preconditioned resid norm 2.667223026209e-05 true
> resid norm 2.490482602536e-04 ||r(i)||/||b|| 3.381169700898e-05
> > > > > > > 18 KSP preconditioned resid norm 2.650813432116e-05 true
> resid norm 2.490473169014e-04 ||r(i)||/||b|| 3.381156893606e-05
> > > > > > > 19 KSP preconditioned resid norm 2.613309555449e-05 true
> resid norm 2.490465633690e-04 ||r(i)||/||b|| 3.381146663375e-05
> > > > > > > 20 KSP preconditioned resid norm 2.644160446804e-05 true
> resid norm 2.490532739949e-04 ||r(i)||/||b|| 3.381237769272e-05
> > > > > > > 21 KSP preconditioned resid norm 2.635987608975e-05 true
> resid norm 2.490499548926e-04 ||r(i)||/||b|| 3.381192707933e-05
> > > > > > > 22 KSP preconditioned resid norm 2.640527129095e-05 true
> resid norm 2.490594066529e-04 ||r(i)||/||b|| 3.381321028466e-05
> > > > > > > 23 KSP preconditioned resid norm 2.627505117691e-05 true
> resid norm 2.490550162585e-04 ||r(i)||/||b|| 3.381261422875e-05
> > > > > > > 24 KSP preconditioned resid norm 2.642659196388e-05 true
> resid norm 2.490504347640e-04 ||r(i)||/||b|| 3.381199222842e-05
> > > > > > > 25 KSP preconditioned resid norm 2.659432190695e-05 true
> resid norm 2.490510775152e-04 ||r(i)||/||b|| 3.381207949065e-05
> > > > > > > 26 KSP preconditioned resid norm 2.687918062951e-05 true
> resid norm 2.490518882015e-04 ||r(i)||/||b|| 3.381218955237e-05
> > > > > > > 27 KSP preconditioned resid norm 2.662909048432e-05 true
> resid norm 2.490446263285e-04 ||r(i)||/||b|| 3.381120365409e-05
> > > > > > > 28 KSP preconditioned resid norm 2.085466483199e-05 true
> resid norm 2.490131612366e-04 ||r(i)||/||b|| 3.380693183886e-05
> > > > > > > 29 KSP preconditioned resid norm 2.098541330282e-05 true
> resid norm 2.490126933398e-04 ||r(i)||/||b|| 3.380686831549e-05
> > > > > > > 30 KSP preconditioned resid norm 2.175345180286e-05 true
> resid norm 2.490098852429e-04 ||r(i)||/||b|| 3.380648707805e-05
> > > > > > > 31 KSP preconditioned resid norm 2.182182437676e-05 true
> resid norm 2.490028301020e-04 ||r(i)||/||b|| 3.380552924648e-05
> > > > > > > 32 KSP preconditioned resid norm 2.152970404369e-05 true
> resid norm 2.490089939838e-04 ||r(i)||/||b|| 3.380636607747e-05
> > > > > > > 33 KSP preconditioned resid norm 2.187932450016e-05 true
> resid norm 2.490085293931e-04 ||r(i)||/||b|| 3.380630300295e-05
> > > > > > > 34 KSP preconditioned resid norm 2.207255875067e-05 true
> resid norm 2.490039036092e-04 ||r(i)||/||b|| 3.380567498971e-05
> > > > > > > 35 KSP preconditioned resid norm 2.205060279701e-05 true
> resid norm 2.490101636150e-04 ||r(i)||/||b|| 3.380652487086e-05
> > > > > > > 36 KSP preconditioned resid norm 2.168654200416e-05 true
> resid norm 2.490091609876e-04 ||r(i)||/||b|| 3.380638875052e-05
> > > > > > > 37 KSP preconditioned resid norm 2.164521042361e-05 true
> resid norm 2.490083143913e-04 ||r(i)||/||b|| 3.380627381352e-05
> > > > > > > 38 KSP preconditioned resid norm 2.154429063973e-05 true
> resid norm 2.490075485470e-04 ||r(i)||/||b|| 3.380616983972e-05
> > > > > > > 39 KSP preconditioned resid norm 2.165962086228e-05 true
> resid norm 2.490099695056e-04 ||r(i)||/||b|| 3.380649851786e-05
> > > > > > > 40 KSP preconditioned resid norm 2.153877616091e-05 true
> resid norm 2.490090652619e-04 ||r(i)||/||b|| 3.380637575444e-05
> > > > > > > 41 KSP preconditioned resid norm 2.347651187611e-05 true
> resid norm 2.490233544624e-04 ||r(i)||/||b|| 3.380831570825e-05
> > > > > > > 42 KSP preconditioned resid norm 2.352860162514e-05 true
> resid norm 2.490191394202e-04 ||r(i)||/||b|| 3.380774345879e-05
> > > > > > > 43 KSP preconditioned resid norm 2.312377506928e-05 true
> resid norm 2.490209491359e-04 ||r(i)||/||b|| 3.380798915237e-05
> > > > > > > 44 KSP preconditioned resid norm 2.295770973533e-05 true
> resid norm 2.490178136759e-04 ||r(i)||/||b|| 3.380756347093e-05
> > > > > > > 45 KSP preconditioned resid norm 2.833646456041e-05 true
> resid norm 2.489991602651e-04 ||r(i)||/||b|| 3.380503101608e-05
> > > > > > > 46 KSP preconditioned resid norm 2.760296424494e-05 true
> resid norm 2.490104320666e-04 ||r(i)||/||b|| 3.380656131682e-05
> > > > > > > 47 KSP preconditioned resid norm 2.451504295239e-05 true
> resid norm 2.490241388672e-04 ||r(i)||/||b|| 3.380842220189e-05
> > > > > > > 48 KSP preconditioned resid norm 2.512391514098e-05 true
> resid norm 2.490245923753e-04 ||r(i)||/||b|| 3.380848377180e-05
> > > > > > > 49 KSP preconditioned resid norm 2.483419450528e-05 true
> resid norm 2.490273364402e-04 ||r(i)||/||b|| 3.380885631602e-05
> > > > > > > 50 KSP preconditioned resid norm 2.507460538466e-05 true
> resid norm 2.490309488780e-04 ||r(i)||/||b|| 3.380934675371e-05
> > > > > > > 51 KSP preconditioned resid norm 2.499708772881e-05 true
> resid norm 2.490300908170e-04 ||r(i)||/||b|| 3.380923026022e-05
> > > > > > > 52 KSP preconditioned resid norm 1.059778259446e-05 true
> resid norm 2.489352833521e-04 ||r(i)||/||b|| 3.379635885420e-05
> > > > > > > 53 KSP preconditioned resid norm 1.074975117060e-05 true
> resid norm 2.489294722901e-04 ||r(i)||/||b|| 3.379556992330e-05
> > > > > > > 54 KSP preconditioned resid norm 1.095242219559e-05 true
> resid norm 2.489295454212e-04 ||r(i)||/||b|| 3.379557985184e-05
> > > > > > > 55 KSP preconditioned resid norm 8.359999674720e-06 true
> resid norm 2.489673581944e-04 ||r(i)||/||b|| 3.380071345137e-05
> > > > > > > 56 KSP preconditioned resid norm 8.368232998470e-06 true
> resid norm 2.489700421343e-04 ||r(i)||/||b|| 3.380107783281e-05
> > > > > > > 57 KSP preconditioned resid norm 8.443378041101e-06 true
> resid norm 2.489702900875e-04 ||r(i)||/||b|| 3.380111149584e-05
> > > > > > > 58 KSP preconditioned resid norm 8.647159584302e-06 true
> resid norm 2.489640805831e-04 ||r(i)||/||b|| 3.380026847095e-05
> > > > > > > 59 KSP preconditioned resid norm 1.024742790737e-05 true
> resid norm 2.489447846660e-04 ||r(i)||/||b|| 3.379764878711e-05
> > > > > > > 60 KSP preconditioned resid norm 1.033394118910e-05 true
> resid norm 2.489441404923e-04 ||r(i)||/||b|| 3.379756133175e-05
> > > > > > > 61 KSP preconditioned resid norm 1.030066336446e-05 true
> resid norm 2.489399918556e-04 ||r(i)||/||b|| 3.379699809776e-05
> > > > > > > 62 KSP preconditioned resid norm 1.029956398963e-05 true
> resid norm 2.489445295139e-04 ||r(i)||/||b|| 3.379761414674e-05
> > > > > > > 63 KSP preconditioned resid norm 1.028190129002e-05 true
> resid norm 2.489456200527e-04 ||r(i)||/||b|| 3.379776220225e-05
> > > > > > > 64 KSP preconditioned resid norm 9.878799185773e-06 true
> resid norm 2.489488742330e-04 ||r(i)||/||b|| 3.379820400160e-05
> > > > > > > 65 KSP preconditioned resid norm 9.917711104174e-06 true
> resid norm 2.489478066593e-04 ||r(i)||/||b|| 3.379805906391e-05
> > > > > > > 66 KSP preconditioned resid norm 1.003572019576e-05 true
> resid norm 2.489441995703e-04 ||r(i)||/||b|| 3.379756935240e-05
> > > > > > > 67 KSP preconditioned resid norm 9.924487278236e-06 true
> resid norm 2.489475403451e-04 ||r(i)||/||b|| 3.379802290812e-05
> > > > > > > 68 KSP preconditioned resid norm 9.804213483359e-06 true
> resid norm 2.489457781760e-04 ||r(i)||/||b|| 3.379778366964e-05
> > > > > > > 69 KSP preconditioned resid norm 9.748922705476e-06 true
> resid norm 2.489408473578e-04 ||r(i)||/||b|| 3.379711424383e-05
> > > > > > > 70 KSP preconditioned resid norm 9.886044523689e-06 true
> resid norm 2.489514438395e-04 ||r(i)||/||b|| 3.379855286071e-05
> > > > > > > 71 KSP preconditioned resid norm 1.083888478689e-05 true
> resid norm 2.489420898851e-04 ||r(i)||/||b|| 3.379728293386e-05
> > > > > > > 72 KSP preconditioned resid norm 1.106561823757e-05 true
> resid norm 2.489364778104e-04 ||r(i)||/||b|| 3.379652101821e-05
> > > > > > > 73 KSP preconditioned resid norm 1.132091515426e-05 true
> resid norm 2.489456804535e-04 ||r(i)||/||b|| 3.379777040248e-05
> > > > > > > 74 KSP preconditioned resid norm 1.330905328963e-05 true
> resid norm 2.489317458981e-04 ||r(i)||/||b|| 3.379587859660e-05
> > > > > > > 75 KSP preconditioned resid norm 1.305555302619e-05 true
> resid norm 2.489320939810e-04 ||r(i)||/||b|| 3.379592585359e-05
> > > > > > > 76 KSP preconditioned resid norm 1.308083397399e-05 true
> resid norm 2.489299951581e-04 ||r(i)||/||b|| 3.379564090977e-05
> > > > > > > 77 KSP preconditioned resid norm 1.320098861853e-05 true
> resid norm 2.489323669317e-04 ||r(i)||/||b|| 3.379596291036e-05
> > > > > > > 78 KSP preconditioned resid norm 1.300160788274e-05 true
> resid norm 2.489306393356e-04 ||r(i)||/||b|| 3.379572836564e-05
> > > > > > > 79 KSP preconditioned resid norm 1.317651537793e-05 true
> resid norm 2.489381364970e-04 ||r(i)||/||b|| 3.379674620752e-05
> > > > > > > 80 KSP preconditioned resid norm 1.309769805765e-05 true
> resid norm 2.489285056062e-04 ||r(i)||/||b|| 3.379543868279e-05
> > > > > > > 81 KSP preconditioned resid norm 1.293686496271e-05 true
> resid norm 2.489347818072e-04 ||r(i)||/||b|| 3.379629076264e-05
> > > > > > > 82 KSP preconditioned resid norm 1.311788285799e-05 true
> resid norm 2.489320040215e-04 ||r(i)||/||b|| 3.379591364037e-05
> > > > > > > 83 KSP preconditioned resid norm 1.313667378798e-05 true
> resid norm 2.489329437217e-04 ||r(i)||/||b|| 3.379604121748e-05
> > > > > > > 84 KSP preconditioned resid norm 1.416138205017e-05 true
> resid norm 2.489266908838e-04 ||r(i)||/||b|| 3.379519230948e-05
> > > > > > > 85 KSP preconditioned resid norm 1.452253464774e-05 true
> resid norm 2.489285688375e-04 ||r(i)||/||b|| 3.379544726729e-05
> > > > > > > 86 KSP preconditioned resid norm 1.426709413370e-05 true
> resid norm 2.489362313402e-04 ||r(i)||/||b|| 3.379648755651e-05
> > > > > > > 87 KSP preconditioned resid norm 1.427480849552e-05 true
> resid norm 2.489378183000e-04 ||r(i)||/||b|| 3.379670300795e-05
> > > > > > > 88 KSP preconditioned resid norm 1.413870980147e-05 true
> resid norm 2.489325756118e-04 ||r(i)||/||b|| 3.379599124153e-05
> > > > > > > 89 KSP preconditioned resid norm 1.353259857657e-05 true
> resid norm 2.489318968308e-04 ||r(i)||/||b|| 3.379589908776e-05
> > > > > > > 90 KSP preconditioned resid norm 1.347676448611e-05 true
> resid norm 2.489332074417e-04 ||r(i)||/||b|| 3.379607702106e-05
> > > > > > > 91 KSP preconditioned resid norm 1.362825902909e-05 true
> resid norm 2.489344974971e-04 ||r(i)||/||b|| 3.379625216367e-05
> > > > > > > 92 KSP preconditioned resid norm 1.346280901052e-05 true
> resid norm 2.489302570131e-04 ||r(i)||/||b|| 3.379567646016e-05
> > > > > > > 93 KSP preconditioned resid norm 1.328052169696e-05 true
> resid norm 2.489346601224e-04 ||r(i)||/||b|| 3.379627424228e-05
> > > > > > > 94 KSP preconditioned resid norm 1.554682082515e-05 true
> resid norm 2.489309078759e-04 ||r(i)||/||b|| 3.379576482365e-05
> > > > > > > 95 KSP preconditioned resid norm 1.557128675775e-05 true
> resid norm 2.489317143582e-04 ||r(i)||/||b|| 3.379587431462e-05
> > > > > > > 96 KSP preconditioned resid norm 1.542571813923e-05 true
> resid norm 2.489319910303e-04 ||r(i)||/||b|| 3.379591187663e-05
> > > > > > > 97 KSP preconditioned resid norm 1.570516684444e-05 true
> resid norm 2.489321980894e-04 ||r(i)||/||b|| 3.379593998772e-05
> > > > > > > 98 KSP preconditioned resid norm 1.600431789899e-05 true
> resid norm 2.489297450311e-04 ||r(i)||/||b|| 3.379560695162e-05
> > > > > > > 99 KSP preconditioned resid norm 1.587495554658e-05 true
> resid norm 2.489339000570e-04 ||r(i)||/||b|| 3.379617105303e-05
> > > > > > > 100 KSP preconditioned resid norm 1.621163002878e-05 true
> resid norm 2.489299953360e-04 ||r(i)||/||b|| 3.379564093392e-05
> > > > > > > 101 KSP preconditioned resid norm 1.627060872574e-05 true
> resid norm 2.489301570161e-04 ||r(i)||/||b|| 3.379566288419e-05
> > > > > > > 102 KSP preconditioned resid norm 1.622931647243e-05 true
> resid norm 2.489277930910e-04 ||r(i)||/||b|| 3.379534194913e-05
> > > > > > > 103 KSP preconditioned resid norm 1.612544300282e-05 true
> resid norm 2.489317483299e-04 ||r(i)||/||b|| 3.379587892674e-05
> > > > > > > 104 KSP preconditioned resid norm 1.880131646630e-05 true
> resid norm 2.489335862583e-04 ||r(i)||/||b|| 3.379612845059e-05
> > > > > > > 105 KSP preconditioned resid norm 1.880563295793e-05 true
> resid norm 2.489365017923e-04 ||r(i)||/||b|| 3.379652427408e-05
> > > > > > > 106 KSP preconditioned resid norm 1.860619184027e-05 true
> resid norm 2.489362382373e-04 ||r(i)||/||b|| 3.379648849288e-05
> > > > > > > 107 KSP preconditioned resid norm 1.877134148719e-05 true
> resid norm 2.489425484523e-04 ||r(i)||/||b|| 3.379734519061e-05
> > > > > > > 108 KSP preconditioned resid norm 1.914810713538e-05 true
> resid norm 2.489347415573e-04 ||r(i)||/||b|| 3.379628529818e-05
> > > > > > > 109 KSP preconditioned resid norm 1.220673255622e-05 true
> resid norm 2.490196186357e-04 ||r(i)||/||b|| 3.380780851884e-05
> > > > > > > 110 KSP preconditioned resid norm 1.215819132910e-05 true
> resid norm 2.490233518072e-04 ||r(i)||/||b|| 3.380831534776e-05
> > > > > > > 111 KSP preconditioned resid norm 1.196565427400e-05 true
> resid norm 2.490279438906e-04 ||r(i)||/||b|| 3.380893878570e-05
> > > > > > > 112 KSP preconditioned resid norm 1.171748185197e-05 true
> resid norm 2.490242578983e-04 ||r(i)||/||b|| 3.380843836198e-05
> > > > > > > 113 KSP preconditioned resid norm 1.162855824118e-05 true
> resid norm 2.490229018536e-04 ||r(i)||/||b|| 3.380825426043e-05
> > > > > > > 114 KSP preconditioned resid norm 1.175594685689e-05 true
> resid norm 2.490274328440e-04 ||r(i)||/||b|| 3.380886940415e-05
> > > > > > > 115 KSP preconditioned resid norm 1.167979454122e-05 true
> resid norm 2.490271161036e-04 ||r(i)||/||b|| 3.380882640232e-05
> > > > > > > 116 KSP preconditioned resid norm 1.181010893019e-05 true
> resid norm 2.490235420657e-04 ||r(i)||/||b|| 3.380834117795e-05
> > > > > > > 117 KSP preconditioned resid norm 1.175206638194e-05 true
> resid norm 2.490263165345e-04 ||r(i)||/||b|| 3.380871784992e-05
> > > > > > > 118 KSP preconditioned resid norm 1.183804125791e-05 true
> resid norm 2.490221353083e-04 ||r(i)||/||b|| 3.380815019145e-05
> > > > > > > 119 KSP preconditioned resid norm 1.186426973727e-05 true
> resid norm 2.490227115336e-04 ||r(i)||/||b|| 3.380822842189e-05
> > > > > > > 120 KSP preconditioned resid norm 1.181986776689e-05 true
> resid norm 2.490257884230e-04 ||r(i)||/||b|| 3.380864615159e-05
> > > > > > > 121 KSP preconditioned resid norm 1.131443277370e-05 true
> resid norm 2.490259110230e-04 ||r(i)||/||b|| 3.380866279620e-05
> > > > > > > 122 KSP preconditioned resid norm 1.114920075859e-05 true
> resid norm 2.490249829382e-04 ||r(i)||/||b|| 3.380853679603e-05
> > > > > > > 123 KSP preconditioned resid norm 1.082073321672e-05 true
> resid norm 2.490314084868e-04 ||r(i)||/||b|| 3.380940915187e-05
> > > > > > > 124 KSP preconditioned resid norm 3.307785860990e-06 true
> resid norm 2.490613501549e-04 ||r(i)||/||b|| 3.381347414155e-05
> > > > > > > 125 KSP preconditioned resid norm 3.287051720572e-06 true
> resid norm 2.490584648195e-04 ||r(i)||/||b|| 3.381308241794e-05
> > > > > > > 126 KSP preconditioned resid norm 3.286797046069e-06 true
> resid norm 2.490654396386e-04 ||r(i)||/||b|| 3.381402934473e-05
> > > > > > > 127 KSP preconditioned resid norm 3.311592899411e-06 true
> resid norm 2.490627973588e-04 ||r(i)||/||b|| 3.381367061922e-05
> > > > > > > 128 KSP preconditioned resid norm 3.560993694635e-06 true
> resid norm 2.490571732816e-04 ||r(i)||/||b|| 3.381290707406e-05
> > > > > > > 129 KSP preconditioned resid norm 3.411994617661e-06 true
> resid norm 2.490652122141e-04 ||r(i)||/||b|| 3.381399846875e-05
> > > > > > > 130 KSP preconditioned resid norm 3.412383310721e-06 true
> resid norm 2.490633454022e-04 ||r(i)||/||b|| 3.381374502359e-05
> > > > > > > 131 KSP preconditioned resid norm 3.288320044878e-06 true
> resid norm 2.490639470096e-04 ||r(i)||/||b|| 3.381382669999e-05
> > > > > > > 132 KSP preconditioned resid norm 3.273215756565e-06 true
> resid norm 2.490640390847e-04 ||r(i)||/||b|| 3.381383920043e-05
> > > > > > > 133 KSP preconditioned resid norm 3.236969051459e-06 true
> resid norm 2.490678216102e-04 ||r(i)||/||b|| 3.381435272985e-05
> > > > > > > 134 KSP preconditioned resid norm 3.203260913942e-06 true
> resid norm 2.490640965346e-04 ||r(i)||/||b|| 3.381384700005e-05
> > > > > > > 135 KSP preconditioned resid norm 3.224117152353e-06 true
> resid norm 2.490655026376e-04 ||r(i)||/||b|| 3.381403789770e-05
> > > > > > > 136 KSP preconditioned resid norm 3.221577997984e-06 true
> resid norm 2.490684737611e-04 ||r(i)||/||b|| 3.381444126823e-05
> > > > > > > 137 KSP preconditioned resid norm 3.195936222128e-06 true
> resid norm 2.490673982333e-04 ||r(i)||/||b|| 3.381429525066e-05
> > > > > > > 138 KSP preconditioned resid norm 3.207528137426e-06 true
> resid norm 2.490641247196e-04 ||r(i)||/||b|| 3.381385082655e-05
> > > > > > > 139 KSP preconditioned resid norm 3.240134271963e-06 true
> resid norm 2.490615861251e-04 ||r(i)||/||b|| 3.381350617773e-05
> > > > > > > 140 KSP preconditioned resid norm 2.698833607230e-06 true
> resid norm 2.490638954889e-04 ||r(i)||/||b|| 3.381381970535e-05
> > > > > > > 141 KSP preconditioned resid norm 2.599151209137e-06 true
> resid norm 2.490657106698e-04 ||r(i)||/||b|| 3.381406614091e-05
> > > > > > > 142 KSP preconditioned resid norm 2.633939920994e-06 true
> resid norm 2.490707754695e-04 ||r(i)||/||b|| 3.381475375652e-05
> > > > > > > 143 KSP preconditioned resid norm 2.519609221376e-06 true
> resid norm 2.490639100480e-04 ||r(i)||/||b|| 3.381382168195e-05
> > > > > > > 144 KSP preconditioned resid norm 3.768526937684e-06 true
> resid norm 2.490654096698e-04 ||r(i)||/||b|| 3.381402527606e-05
> > > > > > > 145 KSP preconditioned resid norm 3.707841943289e-06 true
> resid norm 2.490630207923e-04 ||r(i)||/||b|| 3.381370095336e-05
> > > > > > > 146 KSP preconditioned resid norm 3.698827503486e-06 true
> resid norm 2.490646071561e-04 ||r(i)||/||b|| 3.381391632387e-05
> > > > > > > 147 KSP preconditioned resid norm 3.642747039615e-06 true
> resid norm 2.490610990161e-04 ||r(i)||/||b|| 3.381344004604e-05
> > > > > > > 148 KSP preconditioned resid norm 3.613100087842e-06 true
> resid norm 2.490617159023e-04 ||r(i)||/||b|| 3.381352379676e-05
> > > > > > > 149 KSP preconditioned resid norm 3.637646399299e-06 true
> resid norm 2.490648063023e-04 ||r(i)||/||b|| 3.381394336069e-05
> > > > > > > 150 KSP preconditioned resid norm 3.640235367864e-06 true
> resid norm 2.490648516718e-04 ||r(i)||/||b|| 3.381394952022e-05
> > > > > > > 151 KSP preconditioned resid norm 3.724708848977e-06 true
> resid norm 2.490622201040e-04 ||r(i)||/||b|| 3.381359224901e-05
> > > > > > > 152 KSP preconditioned resid norm 3.665185002770e-06 true
> resid norm 2.490664302790e-04 ||r(i)||/||b|| 3.381416383766e-05
> > > > > > > 153 KSP preconditioned resid norm 3.348992579120e-06 true
> resid norm 2.490655722697e-04 ||r(i)||/||b|| 3.381404735121e-05
> > > > > > > 154 KSP preconditioned resid norm 3.309431137943e-06 true
> resid norm 2.490727563300e-04 ||r(i)||/||b|| 3.381502268535e-05
> > > > > > > 155 KSP preconditioned resid norm 3.299031245428e-06 true
> resid norm 2.490688392843e-04 ||r(i)||/||b|| 3.381449089298e-05
> > > > > > > 156 KSP preconditioned resid norm 3.297127463503e-06 true
> resid norm 2.490642207769e-04 ||r(i)||/||b|| 3.381386386763e-05
> > > > > > > 157 KSP preconditioned resid norm 3.297370198641e-06 true
> resid norm 2.490666651723e-04 ||r(i)||/||b|| 3.381419572764e-05
> > > > > > > 158 KSP preconditioned resid norm 3.290873165210e-06 true
> resid norm 2.490679189538e-04 ||r(i)||/||b|| 3.381436594557e-05
> > > > > > > 159 KSP preconditioned resid norm 3.346705292419e-06 true
> resid norm 2.490617329776e-04 ||r(i)||/||b|| 3.381352611496e-05
> > > > > > > 160 KSP preconditioned resid norm 3.429583550890e-06 true
> resid norm 2.490675116236e-04 ||r(i)||/||b|| 3.381431064494e-05
> > > > > > > 161 KSP preconditioned resid norm 3.425238504679e-06 true
> resid norm 2.490648199058e-04 ||r(i)||/||b|| 3.381394520756e-05
> > > > > > > 162 KSP preconditioned resid norm 3.423484857849e-06 true
> resid norm 2.490723208298e-04 ||r(i)||/||b|| 3.381496356025e-05
> > > > > > > 163 KSP preconditioned resid norm 3.383655922943e-06 true
> resid norm 2.490659981249e-04 ||r(i)||/||b|| 3.381410516686e-05
> > > > > > > 164 KSP preconditioned resid norm 3.477197358452e-06 true
> resid norm 2.490665979073e-04 ||r(i)||/||b|| 3.381418659549e-05
> > > > > > > 165 KSP preconditioned resid norm 3.454672202601e-06 true
> resid norm 2.490651358644e-04 ||r(i)||/||b|| 3.381398810323e-05
> > > > > > > 166 KSP preconditioned resid norm 3.399075522566e-06 true
> resid norm 2.490678159511e-04 ||r(i)||/||b|| 3.381435196154e-05
> > > > > > > 167 KSP preconditioned resid norm 3.305455787400e-06 true
> resid norm 2.490651924523e-04 ||r(i)||/||b|| 3.381399578581e-05
> > > > > > > 168 KSP preconditioned resid norm 3.368445533284e-06 true
> resid norm 2.490688061735e-04 ||r(i)||/||b|| 3.381448639774e-05
> > > > > > > 169 KSP preconditioned resid norm 2.981519724814e-06 true
> resid norm 2.490676378334e-04 ||r(i)||/||b|| 3.381432777964e-05
> > > > > > > 170 KSP preconditioned resid norm 3.034423065539e-06 true
> resid norm 2.490694458885e-04 ||r(i)||/||b|| 3.381457324777e-05
> > > > > > > 171 KSP preconditioned resid norm 2.885972780503e-06 true
> resid norm 2.490688033729e-04 ||r(i)||/||b|| 3.381448601752e-05
> > > > > > > 172 KSP preconditioned resid norm 2.892491075033e-06 true
> resid norm 2.490692993765e-04 ||r(i)||/||b|| 3.381455335678e-05
> > > > > > > 173 KSP preconditioned resid norm 2.921316177611e-06 true
> resid norm 2.490697629787e-04 ||r(i)||/||b|| 3.381461629709e-05
> > > > > > > 174 KSP preconditioned resid norm 2.999889222269e-06 true
> resid norm 2.490707272626e-04 ||r(i)||/||b|| 3.381474721178e-05
> > > > > > > 175 KSP preconditioned resid norm 2.975590207575e-06 true
> resid norm 2.490685439925e-04 ||r(i)||/||b|| 3.381445080310e-05
> > > > > > > 176 KSP preconditioned resid norm 2.983065843597e-06 true
> resid norm 2.490701883671e-04 ||r(i)||/||b|| 3.381467404937e-05
> > > > > > > 177 KSP preconditioned resid norm 2.965959610245e-06 true
> resid norm 2.490711538630e-04 ||r(i)||/||b|| 3.381480512861e-05
> > > > > > > 178 KSP preconditioned resid norm 3.005389788827e-06 true
> resid norm 2.490751808095e-04 ||r(i)||/||b|| 3.381535184150e-05
> > > > > > > 179 KSP preconditioned resid norm 2.956581668772e-06 true
> resid norm 2.490653125636e-04 ||r(i)||/||b|| 3.381401209257e-05
> > > > > > > 180 KSP preconditioned resid norm 2.937498883661e-06 true
> resid norm 2.490666056653e-04 ||r(i)||/||b|| 3.381418764874e-05
> > > > > > > 181 KSP preconditioned resid norm 2.913227475431e-06 true
> resid norm 2.490682436979e-04 ||r(i)||/||b|| 3.381441003402e-05
> > > > > > > 182 KSP preconditioned resid norm 3.048172862254e-06 true
> resid norm 2.490719669872e-04 ||r(i)||/||b|| 3.381491552130e-05
> > > > > > > 183 KSP preconditioned resid norm 3.023868104933e-06 true
> resid norm 2.490648745555e-04 ||r(i)||/||b|| 3.381395262699e-05
> > > > > > > 184 KSP preconditioned resid norm 2.985947506400e-06 true
> resid norm 2.490638818852e-04 ||r(i)||/||b|| 3.381381785846e-05
> > > > > > > 185 KSP preconditioned resid norm 2.840032055776e-06 true
> resid norm 2.490701112392e-04 ||r(i)||/||b|| 3.381466357820e-05
> > > > > > > 186 KSP preconditioned resid norm 2.229279683815e-06 true
> resid norm 2.490609220680e-04 ||r(i)||/||b|| 3.381341602292e-05
> > > > > > > 187 KSP preconditioned resid norm 2.441513276379e-06 true
> resid norm 2.490674056899e-04 ||r(i)||/||b|| 3.381429626300e-05
> > > > > > > 188 KSP preconditioned resid norm 2.467046864016e-06 true
> resid norm 2.490691622632e-04 ||r(i)||/||b|| 3.381453474178e-05
> > > > > > > 189 KSP preconditioned resid norm 2.482124586361e-06 true
> resid norm 2.490664992339e-04 ||r(i)||/||b|| 3.381417319923e-05
> > > > > > > 190 KSP preconditioned resid norm 2.470564926502e-06 true
> resid norm 2.490617019713e-04 ||r(i)||/||b|| 3.381352190543e-05
> > > > > > > 191 KSP preconditioned resid norm 2.457947086578e-06 true
> resid norm 2.490628644250e-04 ||r(i)||/||b|| 3.381367972437e-05
> > > > > > > 192 KSP preconditioned resid norm 2.469444741724e-06 true
> resid norm 2.490639416335e-04 ||r(i)||/||b|| 3.381382597011e-05
> > > > > > > 193 KSP preconditioned resid norm 2.469951525219e-06 true
> resid norm 2.490599769764e-04 ||r(i)||/||b|| 3.381328771385e-05
> > > > > > > 194 KSP preconditioned resid norm 2.467486786643e-06 true
> resid norm 2.490630178622e-04 ||r(i)||/||b|| 3.381370055556e-05
> > > > > > > 195 KSP preconditioned resid norm 2.409684391404e-06 true
> resid norm 2.490640302606e-04 ||r(i)||/||b|| 3.381383800245e-05
> > > > > > > 196 KSP preconditioned resid norm 2.456046691135e-06 true
> resid norm 2.490637730235e-04 ||r(i)||/||b|| 3.381380307900e-05
> > > > > > > 197 KSP preconditioned resid norm 2.300015653805e-06 true
> resid norm 2.490615406913e-04 ||r(i)||/||b|| 3.381350000947e-05
> > > > > > > 198 KSP preconditioned resid norm 2.238328275301e-06 true
> resid norm 2.490647641246e-04 ||r(i)||/||b|| 3.381393763449e-05
> > > > > > > 199 KSP preconditioned resid norm 2.317293820319e-06 true
> resid norm 2.490641611282e-04 ||r(i)||/||b|| 3.381385576951e-05
> > > > > > > 200 KSP preconditioned resid norm 2.359590971314e-06 true
> resid norm 2.490685242974e-04 ||r(i)||/||b|| 3.381444812922e-05
> > > > > > > 201 KSP preconditioned resid norm 2.311199691596e-06 true
> resid norm 2.490656791753e-04 ||r(i)||/||b|| 3.381406186510e-05
> > > > > > > 202 KSP preconditioned resid norm 2.328772904196e-06 true
> resid norm 2.490651045523e-04 ||r(i)||/||b|| 3.381398385220e-05
> > > > > > > 203 KSP preconditioned resid norm 2.332731604717e-06 true
> resid norm 2.490649960574e-04 ||r(i)||/||b|| 3.381396912253e-05
> > > > > > > 204 KSP preconditioned resid norm 2.357629383490e-06 true
> resid norm 2.490686317727e-04 ||r(i)||/||b|| 3.381446272046e-05
> > > > > > > 205 KSP preconditioned resid norm 2.374856180299e-06 true
> resid norm 2.490645897176e-04 ||r(i)||/||b|| 3.381391395637e-05
> > > > > > > 206 KSP preconditioned resid norm 2.340395514404e-06 true
> resid norm 2.490618341127e-04 ||r(i)||/||b|| 3.381353984542e-05
> > > > > > > 207 KSP preconditioned resid norm 2.314963680954e-06 true
> resid norm 2.490676153984e-04 ||r(i)||/||b|| 3.381432473379e-05
> > > > > > > 208 KSP preconditioned resid norm 2.448070953106e-06 true
> resid norm 2.490644606776e-04 ||r(i)||/||b|| 3.381389643743e-05
> > > > > > > 209 KSP preconditioned resid norm 2.428805110632e-06 true
> resid norm 2.490635817597e-04 ||r(i)||/||b|| 3.381377711234e-05
> > > > > > > 210 KSP preconditioned resid norm 2.537929937808e-06 true
> resid norm 2.490680589404e-04 ||r(i)||/||b|| 3.381438495066e-05
> > > > > > > 211 KSP preconditioned resid norm 2.515909029682e-06 true
> resid norm 2.490687803038e-04 ||r(i)||/||b|| 3.381448288557e-05
> > > > > > > 212 KSP preconditioned resid norm 2.497907513266e-06 true
> resid norm 2.490618016885e-04 ||r(i)||/||b|| 3.381353544340e-05
> > > > > > > 213 KSP preconditioned resid norm 1.783501869502e-06 true
> resid norm 2.490632647470e-04 ||r(i)||/||b|| 3.381373407354e-05
> > > > > > > 214 KSP preconditioned resid norm 1.767420653144e-06 true
> resid norm 2.490685569328e-04 ||r(i)||/||b|| 3.381445255992e-05
> > > > > > > 215 KSP preconditioned resid norm 1.854926068272e-06 true
> resid norm 2.490609365464e-04 ||r(i)||/||b|| 3.381341798856e-05
> > > > > > > 216 KSP preconditioned resid norm 1.818308539774e-06 true
> resid norm 2.490639142283e-04 ||r(i)||/||b|| 3.381382224948e-05
> > > > > > > 217 KSP preconditioned resid norm 1.809431578070e-06 true
> resid norm 2.490605125049e-04 ||r(i)||/||b|| 3.381336041915e-05
> > > > > > > 218 KSP preconditioned resid norm 1.789862735999e-06 true
> resid norm 2.490564024901e-04 ||r(i)||/||b|| 3.381280242859e-05
> > > > > > > 219 KSP preconditioned resid norm 1.769239890163e-06 true
> resid norm 2.490647825316e-04 ||r(i)||/||b|| 3.381394013349e-05
> > > > > > > 220 KSP preconditioned resid norm 1.780760773109e-06 true
> resid norm 2.490622606663e-04 ||r(i)||/||b|| 3.381359775589e-05
> > > > > > > 221 KSP preconditioned resid norm 5.009024913368e-07 true
> resid norm 2.490659101637e-04 ||r(i)||/||b|| 3.381409322492e-05
> > > > > > > 222 KSP preconditioned resid norm 4.974450322799e-07 true
> resid norm 2.490714287402e-04 ||r(i)||/||b|| 3.381484244693e-05
> > > > > > > 223 KSP preconditioned resid norm 4.938819481519e-07 true
> resid norm 2.490665661715e-04 ||r(i)||/||b|| 3.381418228693e-05
> > > > > > > 224 KSP preconditioned resid norm 4.973231831266e-07 true
> resid norm 2.490725000995e-04 ||r(i)||/||b|| 3.381498789855e-05
> > > > > > > 225 KSP preconditioned resid norm 5.086864036771e-07 true
> resid norm 2.490664132954e-04 ||r(i)||/||b|| 3.381416153192e-05
> > > > > > > 226 KSP preconditioned resid norm 5.046954570561e-07 true
> resid norm 2.490698772594e-04 ||r(i)||/||b|| 3.381463181226e-05
> > > > > > > 227 KSP preconditioned resid norm 5.086852920874e-07 true
> resid norm 2.490703544723e-04 ||r(i)||/||b|| 3.381469660041e-05
> > > > > > > 228 KSP preconditioned resid norm 5.182381756169e-07 true
> resid norm 2.490665200032e-04 ||r(i)||/||b|| 3.381417601896e-05
> > > > > > > 229 KSP preconditioned resid norm 5.261455182896e-07 true
> resid norm 2.490697169472e-04 ||r(i)||/||b|| 3.381461004770e-05
> > > > > > > 230 KSP preconditioned resid norm 5.265262522400e-07 true
> resid norm 2.490726890541e-04 ||r(i)||/||b|| 3.381501355172e-05
> > > > > > > 231 KSP preconditioned resid norm 5.220652263946e-07 true
> resid norm 2.490689325236e-04 ||r(i)||/||b|| 3.381450355149e-05
> > > > > > > 232 KSP preconditioned resid norm 5.256466259888e-07 true
> resid norm 2.490694033989e-04 ||r(i)||/||b|| 3.381456747923e-05
> > > > > > > 233 KSP preconditioned resid norm 5.443022648374e-07 true
> resid norm 2.490650183144e-04 ||r(i)||/||b|| 3.381397214423e-05
> > > > > > > 234 KSP preconditioned resid norm 5.562619006436e-07 true
> resid norm 2.490764576883e-04 ||r(i)||/||b|| 3.381552519520e-05
> > > > > > > 235 KSP preconditioned resid norm 5.998148629545e-07 true
> resid norm 2.490714032716e-04 ||r(i)||/||b|| 3.381483898922e-05
> > > > > > > 236 KSP preconditioned resid norm 6.498977322955e-07 true
> resid norm 2.490650270144e-04 ||r(i)||/||b|| 3.381397332537e-05
> > > > > > > 237 KSP preconditioned resid norm 6.503686003429e-07 true
> resid norm 2.490706976108e-04 ||r(i)||/||b|| 3.381474318615e-05
> > > > > > > 238 KSP preconditioned resid norm 6.566719023119e-07 true
> resid norm 2.490664107559e-04 ||r(i)||/||b|| 3.381416118714e-05
> > > > > > > 239 KSP preconditioned resid norm 6.549737473208e-07 true
> resid norm 2.490721547909e-04 ||r(i)||/||b|| 3.381494101821e-05
> > > > > > > 240 KSP preconditioned resid norm 6.616898981418e-07 true
> resid norm 2.490679659838e-04 ||r(i)||/||b|| 3.381437233053e-05
> > > > > > > 241 KSP preconditioned resid norm 6.829917691021e-07 true
> resid norm 2.490728328614e-04 ||r(i)||/||b|| 3.381503307553e-05
> > > > > > > 242 KSP preconditioned resid norm 7.030239869389e-07 true
> resid norm 2.490706345115e-04 ||r(i)||/||b|| 3.381473461955e-05
> > > > > > > 243 KSP preconditioned resid norm 7.018435683340e-07 true
> resid norm 2.490650978460e-04 ||r(i)||/||b|| 3.381398294172e-05
> > > > > > > 244 KSP preconditioned resid norm 7.058047080376e-07 true
> resid norm 2.490685975642e-04 ||r(i)||/||b|| 3.381445807618e-05
> > > > > > > 245 KSP preconditioned resid norm 6.896300385099e-07 true
> resid norm 2.490708566380e-04 ||r(i)||/||b|| 3.381476477625e-05
> > > > > > > 246 KSP preconditioned resid norm 7.093960074437e-07 true
> resid norm 2.490667427871e-04 ||r(i)||/||b|| 3.381420626490e-05
> > > > > > > 247 KSP preconditioned resid norm 7.817121711853e-07 true
> resid norm 2.490692299030e-04 ||r(i)||/||b|| 3.381454392480e-05
> > > > > > > 248 KSP preconditioned resid norm 7.976109778309e-07 true
> resid norm 2.490686360729e-04 ||r(i)||/||b|| 3.381446330426e-05
> > > > > > > 249 KSP preconditioned resid norm 7.855322750445e-07 true
> resid norm 2.490720861966e-04 ||r(i)||/||b|| 3.381493170560e-05
> > > > > > > 250 KSP preconditioned resid norm 7.778531114042e-07 true
> resid norm 2.490673235034e-04 ||r(i)||/||b|| 3.381428510506e-05
> > > > > > > 251 KSP preconditioned resid norm 7.848682182070e-07 true
> resid norm 2.490686360729e-04 ||r(i)||/||b|| 3.381446330426e-05
> > > > > > > 252 KSP preconditioned resid norm 7.967291867330e-07 true
> resid norm 2.490724820229e-04 ||r(i)||/||b|| 3.381498544442e-05
> > > > > > > 253 KSP preconditioned resid norm 7.865012959525e-07 true
> resid norm 2.490666662028e-04 ||r(i)||/||b|| 3.381419586754e-05
> > > > > > > 254 KSP preconditioned resid norm 7.656025385804e-07 true
> resid norm 2.490686283214e-04 ||r(i)||/||b|| 3.381446225190e-05
> > > > > > > 255 KSP preconditioned resid norm 7.757018653468e-07 true
> resid norm 2.490655983763e-04 ||r(i)||/||b|| 3.381405089553e-05
> > > > > > > 256 KSP preconditioned resid norm 6.686490372981e-07 true
> resid norm 2.490715698964e-04 ||r(i)||/||b|| 3.381486161081e-05
> > > > > > > 257 KSP preconditioned resid norm 6.596005109428e-07 true
> resid norm 2.490666403003e-04 ||r(i)||/||b|| 3.381419235092e-05
> > > > > > > 258 KSP preconditioned resid norm 6.681742296333e-07 true
> resid norm 2.490683725835e-04 ||r(i)||/||b|| 3.381442753198e-05
> > > > > > > 259 KSP preconditioned resid norm 1.089245482033e-06 true
> resid norm 2.490688086568e-04 ||r(i)||/||b|| 3.381448673488e-05
> > > > > > > 260 KSP preconditioned resid norm 1.099844873189e-06 true
> resid norm 2.490690703265e-04 ||r(i)||/||b|| 3.381452226011e-05
> > > > > > > 261 KSP preconditioned resid norm 1.112925540869e-06 true
> resid norm 2.490664481058e-04 ||r(i)||/||b|| 3.381416625790e-05
> > > > > > > 262 KSP preconditioned resid norm 1.113056910480e-06 true
> resid norm 2.490658753273e-04 ||r(i)||/||b|| 3.381408849541e-05
> > > > > > > 263 KSP preconditioned resid norm 1.104801535149e-06 true
> resid norm 2.490736510776e-04 ||r(i)||/||b|| 3.381514415953e-05
> > > > > > > 264 KSP preconditioned resid norm 1.158709147873e-06 true
> resid norm 2.490607531152e-04 ||r(i)||/||b|| 3.381339308528e-05
> > > > > > > 265 KSP preconditioned resid norm 1.178985740182e-06 true
> resid norm 2.490727895619e-04 ||r(i)||/||b|| 3.381502719703e-05
> > > > > > > 266 KSP preconditioned resid norm 1.165130533478e-06 true
> resid norm 2.490639076693e-04 ||r(i)||/||b|| 3.381382135901e-05
> > > > > > > 267 KSP preconditioned resid norm 1.181364114499e-06 true
> resid norm 2.490667871436e-04 ||r(i)||/||b|| 3.381421228690e-05
> > > > > > > 268 KSP preconditioned resid norm 1.170295348543e-06 true
> resid norm 2.490662613306e-04 ||r(i)||/||b|| 3.381414090063e-05
> > > > > > > 269 KSP preconditioned resid norm 1.213243016230e-06 true
> resid norm 2.490666173719e-04 ||r(i)||/||b|| 3.381418923808e-05
> > > > > > > 270 KSP preconditioned resid norm 1.239691953997e-06 true
> resid norm 2.490678323197e-04 ||r(i)||/||b|| 3.381435418381e-05
> > > > > > > 271 KSP preconditioned resid norm 1.219891740100e-06 true
> resid norm 2.490625009256e-04 ||r(i)||/||b|| 3.381363037437e-05
> > > > > > > 272 KSP preconditioned resid norm 1.231321334346e-06 true
> resid norm 2.490659733696e-04 ||r(i)||/||b|| 3.381410180599e-05
> > > > > > > 273 KSP preconditioned resid norm 1.208183234158e-06 true
> resid norm 2.490685987255e-04 ||r(i)||/||b|| 3.381445823385e-05
> > > > > > > 274 KSP preconditioned resid norm 1.211768545589e-06 true
> resid norm 2.490671548953e-04 ||r(i)||/||b|| 3.381426221421e-05
> > > > > > > 275 KSP preconditioned resid norm 1.209433459842e-06 true
> resid norm 2.490669016096e-04 ||r(i)||/||b|| 3.381422782722e-05
> > > > > > > 276 KSP preconditioned resid norm 1.223729184405e-06 true
> resid norm 2.490658128014e-04 ||r(i)||/||b|| 3.381408000666e-05
> > > > > > > 277 KSP preconditioned resid norm 1.243915201868e-06 true
> resid norm 2.490693375756e-04 ||r(i)||/||b|| 3.381455854282e-05
> > > > > > > 278 KSP preconditioned resid norm 1.231994655529e-06 true
> resid norm 2.490682988311e-04 ||r(i)||/||b|| 3.381441751910e-05
> > > > > > > 279 KSP preconditioned resid norm 1.227930683777e-06 true
> resid norm 2.490667825866e-04 ||r(i)||/||b|| 3.381421166823e-05
> > > > > > > 280 KSP preconditioned resid norm 1.193458846469e-06 true
> resid norm 2.490687366117e-04 ||r(i)||/||b|| 3.381447695378e-05
> > > > > > > 281 KSP preconditioned resid norm 1.217089059805e-06 true
> resid norm 2.490674797371e-04 ||r(i)||/||b|| 3.381430631591e-05
> > > > > > > 282 KSP preconditioned resid norm 1.249318287709e-06 true
> resid norm 2.490662866951e-04 ||r(i)||/||b|| 3.381414434420e-05
> > > > > > > 283 KSP preconditioned resid norm 1.183320029547e-06 true
> resid norm 2.490645783630e-04 ||r(i)||/||b|| 3.381391241482e-05
> > > > > > > 284 KSP preconditioned resid norm 1.174730603102e-06 true
> resid norm 2.490686881647e-04 ||r(i)||/||b|| 3.381447037643e-05
> > > > > > > 285 KSP preconditioned resid norm 1.175838261923e-06 true
> resid norm 2.490665969300e-04 ||r(i)||/||b|| 3.381418646281e-05
> > > > > > > 286 KSP preconditioned resid norm 1.188946188368e-06 true
> resid norm 2.490661974622e-04 ||r(i)||/||b|| 3.381413222961e-05
> > > > > > > 287 KSP preconditioned resid norm 1.177848565707e-06 true
> resid norm 2.490660236206e-04 ||r(i)||/||b|| 3.381410862824e-05
> > > > > > > 288 KSP preconditioned resid norm 1.200075508281e-06 true
> resid norm 2.490645353536e-04 ||r(i)||/||b|| 3.381390657571e-05
> > > > > > > 289 KSP preconditioned resid norm 1.184589570618e-06 true
> resid norm 2.490664920355e-04 ||r(i)||/||b|| 3.381417222195e-05
> > > > > > > 290 KSP preconditioned resid norm 1.221114703873e-06 true
> resid norm 2.490670597538e-04 ||r(i)||/||b|| 3.381424929746e-05
> > > > > > > 291 KSP preconditioned resid norm 1.249479658256e-06 true
> resid norm 2.490641582876e-04 ||r(i)||/||b|| 3.381385538385e-05
> > > > > > > 292 KSP preconditioned resid norm 1.245768496850e-06 true
> resid norm 2.490704480588e-04 ||r(i)||/||b|| 3.381470930606e-05
> > > > > > > 293 KSP preconditioned resid norm 1.243742607953e-06 true
> resid norm 2.490649690604e-04 ||r(i)||/||b|| 3.381396545733e-05
> > > > > > > 294 KSP preconditioned resid norm 1.342758483339e-06 true
> resid norm 2.490676207432e-04 ||r(i)||/||b|| 3.381432545942e-05
> > > > > > > 295 KSP preconditioned resid norm 1.353816099600e-06 true
> resid norm 2.490695263153e-04 ||r(i)||/||b|| 3.381458416681e-05
> > > > > > > 296 KSP preconditioned resid norm 1.343886763293e-06 true
> resid norm 2.490673674307e-04 ||r(i)||/||b|| 3.381429106879e-05
> > > > > > > 297 KSP preconditioned resid norm 1.355511022815e-06 true
> resid norm 2.490686565995e-04 ||r(i)||/||b|| 3.381446609103e-05
> > > > > > > 298 KSP preconditioned resid norm 1.347247627243e-06 true
> resid norm 2.490696287707e-04 ||r(i)||/||b|| 3.381459807652e-05
> > > > > > > 299 KSP preconditioned resid norm 1.414742595618e-06 true
> resid norm 2.490749815091e-04 ||r(i)||/||b|| 3.381532478374e-05
> > > > > > > 300 KSP preconditioned resid norm 1.418560683189e-06 true
> resid norm 2.490721501153e-04 ||r(i)||/||b|| 3.381494038343e-05
> > > > > > > 301 KSP preconditioned resid norm 1.416276404923e-06 true
> resid norm 2.490689576447e-04 ||r(i)||/||b|| 3.381450696203e-05
> > > > > > > 302 KSP preconditioned resid norm 1.431448272112e-06 true
> resid norm 2.490688812701e-04 ||r(i)||/||b|| 3.381449659312e-05
> > > > > > > 303 KSP preconditioned resid norm 1.446154958969e-06 true
> resid norm 2.490727536322e-04 ||r(i)||/||b|| 3.381502231909e-05
> > > > > > > 304 KSP preconditioned resid norm 1.468860617921e-06 true
> resid norm 2.490692363788e-04 ||r(i)||/||b|| 3.381454480397e-05
> > > > > > > 305 KSP preconditioned resid norm 1.627595214971e-06 true
> resid norm 2.490687019603e-04 ||r(i)||/||b|| 3.381447224938e-05
> > > > > > > 306 KSP preconditioned resid norm 1.614384672893e-06 true
> resid norm 2.490687019603e-04 ||r(i)||/||b|| 3.381447224938e-05
> > > > > > > 307 KSP preconditioned resid norm 1.605568020532e-06 true
> resid norm 2.490699757693e-04 ||r(i)||/||b|| 3.381464518632e-05
> > > > > > > 308 KSP preconditioned resid norm 1.617069685075e-06 true
> resid norm 2.490649282923e-04 ||r(i)||/||b|| 3.381395992249e-05
> > > > > > > 309 KSP preconditioned resid norm 1.654297792738e-06 true
> resid norm 2.490644766626e-04 ||r(i)||/||b|| 3.381389860760e-05
> > > > > > > 310 KSP preconditioned resid norm 1.587528143215e-06 true
> resid norm 2.490696752096e-04 ||r(i)||/||b|| 3.381460438124e-05
> > > > > > > 311 KSP preconditioned resid norm 1.662782022388e-06 true
> resid norm 2.490699317737e-04 ||r(i)||/||b|| 3.381463921332e-05
> > > > > > > 312 KSP preconditioned resid norm 1.618211471748e-06 true
> resid norm 2.490735831308e-04 ||r(i)||/||b|| 3.381513493483e-05
> > > > > > > 313 KSP preconditioned resid norm 1.609074961921e-06 true
> resid norm 2.490679566436e-04 ||r(i)||/||b|| 3.381437106247e-05
> > > > > > > 314 KSP preconditioned resid norm 1.548068942878e-06 true
> resid norm 2.490660071226e-04 ||r(i)||/||b|| 3.381410638842e-05
> > > > > > > 315 KSP preconditioned resid norm 1.526718322150e-06 true
> resid norm 2.490619832967e-04 ||r(i)||/||b|| 3.381356009919e-05
> > > > > > > 316 KSP preconditioned resid norm 1.553150959105e-06 true
> resid norm 2.490660071226e-04 ||r(i)||/||b|| 3.381410638842e-05
> > > > > > > 317 KSP preconditioned resid norm 1.615015320906e-06 true
> resid norm 2.490672348079e-04 ||r(i)||/||b|| 3.381427306343e-05
> > > > > > > 318 KSP preconditioned resid norm 1.602904469797e-06 true
> resid norm 2.490696731006e-04 ||r(i)||/||b|| 3.381460409491e-05
> > > > > > > 319 KSP preconditioned resid norm 1.538140323073e-06 true
> resid norm 2.490722982494e-04 ||r(i)||/||b|| 3.381496049466e-05
> > > > > > > 320 KSP preconditioned resid norm 1.534779679430e-06 true
> resid norm 2.490778499789e-04 ||r(i)||/||b|| 3.381571421763e-05
> > > > > > > 321 KSP preconditioned resid norm 1.547155843355e-06 true
> resid norm 2.490767612985e-04 ||r(i)||/||b|| 3.381556641442e-05
> > > > > > > 322 KSP preconditioned resid norm 1.422137008870e-06 true
> resid norm 2.490737676309e-04 ||r(i)||/||b|| 3.381515998323e-05
> > > > > > > 323 KSP preconditioned resid norm 1.403072558954e-06 true
> resid norm 2.490741361870e-04 ||r(i)||/||b|| 3.381521001975e-05
> > > > > > > 324 KSP preconditioned resid norm 1.373070436118e-06 true
> resid norm 2.490742214990e-04 ||r(i)||/||b|| 3.381522160202e-05
> > > > > > > 325 KSP preconditioned resid norm 1.359547585233e-06 true
> resid norm 2.490792987570e-04 ||r(i)||/||b|| 3.381591090902e-05
> > > > > > > 326 KSP preconditioned resid norm 1.370351913612e-06 true
> resid norm 2.490727161158e-04 ||r(i)||/||b|| 3.381501722573e-05
> > > > > > > 327 KSP preconditioned resid norm 1.365238666187e-06 true
> resid norm 2.490716949642e-04 ||r(i)||/||b|| 3.381487859046e-05
> > > > > > > 328 KSP preconditioned resid norm 1.369073373042e-06 true
> resid norm 2.490807360288e-04 ||r(i)||/||b|| 3.381610603826e-05
> > > > > > > 329 KSP preconditioned resid norm 1.426698981572e-06 true
> resid norm 2.490791479521e-04 ||r(i)||/||b|| 3.381589043520e-05
> > > > > > > 330 KSP preconditioned resid norm 1.445542403570e-06 true
> resid norm 2.490775981409e-04 ||r(i)||/||b|| 3.381568002720e-05
> > > > > > > 331 KSP preconditioned resid norm 1.464506963984e-06 true
> resid norm 2.490740562430e-04 ||r(i)||/||b|| 3.381519916626e-05
> > > > > > > 332 KSP preconditioned resid norm 1.461462964401e-06 true
> resid norm 2.490768016856e-04 ||r(i)||/||b|| 3.381557189753e-05
> > > > > > > 333 KSP preconditioned resid norm 1.476680847971e-06 true
> resid norm 2.490744321516e-04 ||r(i)||/||b|| 3.381525020097e-05
> > > > > > > 334 KSP preconditioned resid norm 1.459640372198e-06 true
> resid norm 2.490788817993e-04 ||r(i)||/||b|| 3.381585430132e-05
> > > > > > > 335 KSP preconditioned resid norm 1.790770882365e-06 true
> resid norm 2.490771711471e-04 ||r(i)||/||b|| 3.381562205697e-05
> > > > > > > 336 KSP preconditioned resid norm 1.803770155018e-06 true
> resid norm 2.490768953858e-04 ||r(i)||/||b|| 3.381558461860e-05
> > > > > > > 337 KSP preconditioned resid norm 1.787821255995e-06 true
> resid norm 2.490767985676e-04 ||r(i)||/||b|| 3.381557147421e-05
> > > > > > > 338 KSP preconditioned resid norm 1.749912220831e-06 true
> resid norm 2.490760198704e-04 ||r(i)||/||b|| 3.381546575545e-05
> > > > > > > 339 KSP preconditioned resid norm 1.802915839010e-06 true
> resid norm 2.490815556273e-04 ||r(i)||/||b|| 3.381621730993e-05
> > > > > > > 340 KSP preconditioned resid norm 1.800777670709e-06 true
> resid norm 2.490823909286e-04 ||r(i)||/||b|| 3.381633071347e-05
> > > > > > > 341 KSP preconditioned resid norm 1.962516327690e-06 true
> resid norm 2.490773477410e-04 ||r(i)||/||b|| 3.381564603199e-05
> > > > > > > 342 KSP preconditioned resid norm 1.981726465132e-06 true
> resid norm 2.490769116884e-04 ||r(i)||/||b|| 3.381558683191e-05
> > > > > > > 343 KSP preconditioned resid norm 1.963419167052e-06 true
> resid norm 2.490764009914e-04 ||r(i)||/||b|| 3.381551749783e-05
> > > > > > > 344 KSP preconditioned resid norm 1.992082169278e-06 true
> resid norm 2.490806829883e-04 ||r(i)||/||b|| 3.381609883728e-05
> > > > > > > 345 KSP preconditioned resid norm 1.981005134253e-06 true
> resid norm 2.490748677339e-04 ||r(i)||/||b|| 3.381530933721e-05
> > > > > > > 346 KSP preconditioned resid norm 1.959802663114e-06 true
> resid norm 2.490773752317e-04 ||r(i)||/||b|| 3.381564976423e-05
> > > > > > >
> > > > > > > On Sat, Oct 21, 2017 at 5:25 PM, Matthew Knepley <
> knepley at gmail.com> wrote:
> > > > > > > On Sat, Oct 21, 2017 at 5:21 PM, Hao Zhang <
> hbcbh1999 at gmail.com> wrote:
> > > > > > > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
> > > > > > > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
> > > > > > >
> > > > > > > ierr = VecAssemblyBegin(x);
> > > > > > > ierr = VecAssemblyEnd(x);
> > > > > > > This is probably unnecessary
> > > > > > >
> > > > > > > ierr = VecAssemblyBegin(b);
> > > > > > > ierr = VecAssemblyEnd(b);
> > > > > > > This is probably unnecessary
> > > > > > >
> > > > > > >
> > > > > > > ierr = MatNullSpaceCreate(PETSC_COMM_
> WORLD,PETSC_TRUE,0,PETSC_NULL,&nullsp);
> > > > > > > ierr = MatSetNullSpace(A,nullsp); // Petsc-3.8
> > > > > > > Is your rhs consistent with this nullspace?
> > > > > > >
> > > > > > > // KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);
> > > > > > > KSPSetOperators(ksp,A,A);
> > > > > > >
> > > > > > > KSPSetType(ksp,KSPBCGS);
> > > > > > >
> > > > > > > KSPSetComputeSingularValues(ksp, PETSC_TRUE);
> > > > > > > #if defined(__HYPRE__)
> > > > > > > KSPGetPC(ksp, &pc);
> > > > > > > PCSetType(pc, PCHYPRE);
> > > > > > > PCHYPRESetType(pc,"boomeramg");
> > > > > > > This is terribly unnecessary. You just use
> > > > > > >
> > > > > > > -pc_type hypre -pc_hypre_type boomeramg
> > > > > > >
> > > > > > > or
> > > > > > >
> > > > > > > -pc_type gamg
> > > > > > >
> > > > > > > #else
> > > > > > > KSPSetType(ksp,KSPBCGSL);
> > > > > > > KSPBCGSLSetEll(ksp,2);
> > > > > > > #endif /* defined(__HYPRE__) */
> > > > > > >
> > > > > > > KSPSetFromOptions(ksp);
> > > > > > > KSPSetUp(ksp);
> > > > > > >
> > > > > > > ierr = KSPSolve(ksp,b,x);
> > > > > > >
> > > > > > >
> > > > > > > command line
> > > > > > >
> > > > > > > You did not provide any of what I asked for the in the
> eprevious mail.
> > > > > > >
> > > > > > > Matt
> > > > > > >
> > > > > > > On Sat, Oct 21, 2017 at 5:16 PM, Matthew Knepley <
> knepley at gmail.com> wrote:
> > > > > > > On Sat, Oct 21, 2017 at 5:04 PM, Hao Zhang <
> hbcbh1999 at gmail.com> wrote:
> > > > > > > hi,
> > > > > > >
> > > > > > > I implemented HYPRE preconditioner for my study due to the
> fact that without preconditioner, PETSc solver will take thousands of
> iterations to converge for fine grid simulation.
> > > > > > >
> > > > > > > with HYPRE, depending on the parallel partition, it will take
> HYPRE forever to do anything. observation of output file is that the
> simulation is hanging with no output.
> > > > > > >
> > > > > > > Any idea what happened? will post snippet of code.
> > > > > > >
> > > > > > > 1) For any question about convergence, we need to see the
> output of
> > > > > > >
> > > > > > > -ksp_view_pre -ksp_view -ksp_monitor_true_residual
> -ksp_converged_reason
> > > > > > >
> > > > > > > 2) Hypre has many preconditioners, which one are you talking
> about
> > > > > > >
> > > > > > > 3) PETSc has some preconditioners in common with Hypre, like
> AMG
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Matt
> > > > > > >
> > > > > > > --
> > > > > > > Hao Zhang
> > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > Stony Brook University,
> > > > > > > Stony Brook, New York, 11790
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > What most experimenters take for granted before they begin
> their experiments is infinitely more interesting than any results to which
> their experiments lead.
> > > > > > > -- Norbert Wiener
> > > > > > >
> > > > > > > https://www.cse.buffalo.edu/~knepley/
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Hao Zhang
> > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > Stony Brook University,
> > > > > > > Stony Brook, New York, 11790
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > What most experimenters take for granted before they begin
> their experiments is infinitely more interesting than any results to which
> their experiments lead.
> > > > > > > -- Norbert Wiener
> > > > > > >
> > > > > > > https://www.cse.buffalo.edu/~knepley/
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Hao Zhang
> > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > Stony Brook University,
> > > > > > > Stony Brook, New York, 11790
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Hao Zhang
> > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > Stony Brook University,
> > > > > > > Stony Brook, New York, 11790
> > > > > >
> > > > > > --
> > > > > > Hao Zhang
> > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > Stony Brook University,
> > > > > > Stony Brook, New York, 11790
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Hao Zhang
> > > > > Dept. of Applid Mathematics and Statistics,
> > > > > Stony Brook University,
> > > > > Stony Brook, New York, 11790
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Hao Zhang
> > > > Dept. of Applid Mathematics and Statistics,
> > > > Stony Brook University,
> > > > Stony Brook, New York, 11790
> > >
> > >
> > >
> > >
> > > --
> > > Hao Zhang
> > > Dept. of Applid Mathematics and Statistics,
> > > Stony Brook University,
> > > Stony Brook, New York, 11790
> >
> >
> >
> >
> > --
> > Hao Zhang
> > Dept. of Applid Mathematics and Statistics,
> > Stony Brook University,
> > Stony Brook, New York, 11790
>
>
--
Hao Zhang
Dept. of Applid Mathematics and Statistics,
Stony Brook University,
Stony Brook, New York, 11790
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171022/23c67a88/attachment-0001.html>
More information about the petsc-users
mailing list