[petsc-users] HYPRE hanging or slow? from observation
Mark Adams
mfadams at lbl.gov
Mon Oct 23 08:01:18 CDT 2017
Just to be clear: 1) are you solving the Laplacian (div grad) and 2) what
type of discretizations are you using? and 3) do you have stretched or
terrible grids in some way?
On Sun, Oct 22, 2017 at 3:57 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> One thing important to understand is that multigrid is an optimal or
> nearly optimal algorithm. This means, when it works, as you refine the mesh
> the number of iterations remains nearly constant, regardless of the problem
> size and number of processes. Simple preconditioners such as ILU, block
> Jacobi, one level additive Schwarz etc have iterations that increase with
> the problem size and likely also with the number of processes. Thus these
> algorithms become essentially impractical for very large problems while
> multigrid can remain practical (when it works).
>
> Good luck
>
> Barry
> > On Oct 22, 2017, at 2:35 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> >
> > Thanks for all the inputs. before simulating finer grid, HYPRE wasn't
> used and simulations were just fine. I will do a few tests and post more
> information later.
> >
> > On Sun, Oct 22, 2017 at 12:11 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >
> > > On Oct 21, 2017, at 11:16 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > >
> > > the reason is when I do finer grid simulation, matrix become more
> stiff.
> >
> > Are you saying that for a finer grid but everything else the same,
> the convergence of hypre (with the same GMRES) with the same options gets
> much worse? This normally will not happen, that is the fundamental beauty
> of multigrid methods (when they work well).
> >
> > Yes the matrix condition number increases but multigrid doesn't care
> about that, its number of iterations should remain pretty much the same.
> >
> > Something must be different (with this finer grid case), either the
> mesh becomes horrible, or the physics changes, or there are errors in the
> code that lead to the problem.
> >
> > What happens if you just refine the mesh a little? Then a little
> more? Then a little more? Does the convergence rate suddenly go bad at some
> point, or does it just get worse slowly?
> >
> > Barry
> >
> >
> >
> > > Much larger condition number. just to give you a perspective, it will
> take 6000 iterations to converge and the solver does converge. I want to
> reduce the number of iterations while keeping the convergence rate. that's
> main drive to do so much heavy lifting around. please advise. snippet will
> be provided upon request.
> > >
> > > Thanks again.
> > >
> > > On Sun, Oct 22, 2017 at 12:08 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > > Oh, you change KSP but not hypre. I did not understand this.
> > >
> > > Why not just use GMRES all the time? Why mess with BCGS if it is not
> robust? Not worth the small optimization if it breaks everything.
> > >
> > > Barry
> > >
> > >
> > >
> > > > On Oct 21, 2017, at 11:05 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > >
> > > > this is the initial pressure solver output regarding use of PETSc.
> it failed to converge after 40000 iterations, then use GMRES.
> > > >
> > > > 39987 KSP preconditioned resid norm 3.853125986269e-08 true resid
> norm 1.147359212216e-05 ||r(i)||/||b|| 1.557696568706e-06
> > > > 39988 KSP preconditioned resid norm 3.853126044003e-08 true resid
> norm 1.147359257282e-05 ||r(i)||/||b|| 1.557696629889e-06
> > > > 39989 KSP preconditioned resid norm 3.853126052100e-08 true resid
> norm 1.147359233695e-05 ||r(i)||/||b|| 1.557696597866e-06
> > > > 39990 KSP preconditioned resid norm 3.853126027357e-08 true resid
> norm 1.147359219860e-05 ||r(i)||/||b|| 1.557696579083e-06
> > > > 39991 KSP preconditioned resid norm 3.853126058478e-08 true resid
> norm 1.147359234281e-05 ||r(i)||/||b|| 1.557696598662e-06
> > > > 39992 KSP preconditioned resid norm 3.853126064006e-08 true resid
> norm 1.147359261420e-05 ||r(i)||/||b|| 1.557696635506e-06
> > > > 39993 KSP preconditioned resid norm 3.853126050203e-08 true resid
> norm 1.147359235972e-05 ||r(i)||/||b|| 1.557696600957e-06
> > > > 39994 KSP preconditioned resid norm 3.853126050182e-08 true resid
> norm 1.147359253713e-05 ||r(i)||/||b|| 1.557696625043e-06
> > > > 39995 KSP preconditioned resid norm 3.853125976795e-08 true resid
> norm 1.147359226222e-05 ||r(i)||/||b|| 1.557696587720e-06
> > > > 39996 KSP preconditioned resid norm 3.853125805127e-08 true resid
> norm 1.147359262747e-05 ||r(i)||/||b|| 1.557696637308e-06
> > > > 39997 KSP preconditioned resid norm 3.853125811756e-08 true resid
> norm 1.147359216008e-05 ||r(i)||/||b|| 1.557696573853e-06
> > > > 39998 KSP preconditioned resid norm 3.853125827833e-08 true resid
> norm 1.147359238372e-05 ||r(i)||/||b|| 1.557696604216e-06
> > > > 39999 KSP preconditioned resid norm 3.853127937068e-08 true resid
> norm 1.147359264043e-05 ||r(i)||/||b|| 1.557696639067e-06
> > > > 40000 KSP preconditioned resid norm 3.853122732867e-08 true resid
> norm 1.147359257776e-05 ||r(i)||/||b|| 1.557696630559e-06
> > > > Linear solve did not converge due to DIVERGED_ITS iterations 40000
> > > > KSP Object: 24 MPI processes
> > > > type: bcgs
> > > > maximum iterations=40000, initial guess is zero
> > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > left preconditioning
> > > > using PRECONDITIONED norm type for convergence test
> > > > PC Object: 24 MPI processes
> > > > type: hypre
> > > > HYPRE BoomerAMG preconditioning
> > > > Cycle type V
> > > > Maximum number of levels 25
> > > > Maximum number of iterations PER hypre call 1
> > > > Convergence tolerance PER hypre call 0.
> > > > Threshold for strong coupling 0.25
> > > > Interpolation truncation factor 0.
> > > > Interpolation: max elements per row 0
> > > > Number of levels of aggressive coarsening 0
> > > > Number of paths for aggressive coarsening 1
> > > > Maximum row sums 0.9
> > > > Sweeps down 1
> > > > Sweeps up 1
> > > > Sweeps on coarse 1
> > > > Relax down symmetric-SOR/Jacobi
> > > > Relax up symmetric-SOR/Jacobi
> > > > Relax on coarse Gaussian-elimination
> > > > Relax weight (all) 1.
> > > > Outer relax weight (all) 1.
> > > > Using CF-relaxation
> > > > Not using more complex smoothers.
> > > > Measure type local
> > > > Coarsen type Falgout
> > > > Interpolation type classical
> > > > linear system matrix = precond matrix:
> > > > Mat Object: A 24 MPI processes
> > > > type: mpiaij
> > > > rows=497664, cols=497664
> > > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > > total number of mallocs used during MatSetValues calls =0
> > > > has attached null space
> > > > not using I-node (on process 0) routines
> > > >
> > > > The solution diverges for p0! The residual is 3.853123e-08. Solve
> again using GMRES!
> > > > KSP Object: 24 MPI processes
> > > > type: gmres
> > > > restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > > > happy breakdown tolerance 1e-30
> > > > maximum iterations=40000, initial guess is zero
> > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > left preconditioning
> > > > using PRECONDITIONED norm type for convergence test
> > > > PC Object: 24 MPI processes
> > > > type: hypre
> > > > HYPRE BoomerAMG preconditioning
> > > > Cycle type V
> > > > Maximum number of levels 25
> > > > Maximum number of iterations PER hypre call 1
> > > > Convergence tolerance PER hypre call 0.
> > > > Threshold for strong coupling 0.25
> > > > Interpolation truncation factor 0.
> > > > Interpolation: max elements per row 0
> > > > Number of levels of aggressive coarsening 0
> > > > Number of paths for aggressive coarsening 1
> > > > Maximum row sums 0.9
> > > > Sweeps down 1
> > > > Sweeps up 1
> > > > Sweeps on coarse 1
> > > > Relax down symmetric-SOR/Jacobi
> > > > Relax up symmetric-SOR/Jacobi
> > > > Relax on coarse Gaussian-elimination
> > > > Relax weight (all) 1.
> > > > Outer relax weight (all) 1.
> > > > Using CF-relaxation
> > > > Not using more complex smoothers.
> > > > Measure type local
> > > > Coarsen type Falgout
> > > > Interpolation type classical
> > > > linear system matrix = precond matrix:
> > > > Mat Object: A 24 MPI processes
> > > > type: mpiaij
> > > > rows=497664, cols=497664
> > > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > > total number of mallocs used during MatSetValues calls =0
> > > > has attached null space
> > > > not using I-node (on process 0) routines
> > > > 0 KSP preconditioned resid norm 1.593802941804e+01 true resid norm
> 7.365742695119e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > > 1 KSP preconditioned resid norm 6.338666661133e-01 true resid norm
> 2.880722209358e+00 ||r(i)||/||b|| 3.910973174867e-01
> > > > 2 KSP preconditioned resid norm 3.913420828350e-02 true resid norm
> 9.544089760671e-01 ||r(i)||/||b|| 1.295740315093e-01
> > > > 3 KSP preconditioned resid norm 2.928070366435e-03 true resid norm
> 1.874294004628e-01 ||r(i)||/||b|| 2.544609664236e-02
> > > > 4 KSP preconditioned resid norm 2.165607525823e-04 true resid norm
> 3.121122463949e-02 ||r(i)||/||b|| 4.237349298146e-03
> > > > 5 KSP preconditioned resid norm 1.635476480407e-05 true resid norm
> 3.984315313831e-03 ||r(i)||/||b|| 5.409251284967e-04
> > > > 6 KSP preconditioned resid norm 1.283358350575e-06 true resid norm
> 4.566583802915e-04 ||r(i)||/||b|| 6.199760149022e-05
> > > > 7 KSP preconditioned resid norm 8.479469225747e-08 true resid norm
> 3.824581791112e-05 ||r(i)||/||b|| 5.192391248810e-06
> > > > 8 KSP preconditioned resid norm 5.358636504683e-09 true resid norm
> 2.829730442033e-06 ||r(i)||/||b|| 3.841744898187e-07
> > > > 9 KSP preconditioned resid norm 3.447874504193e-10 true resid norm
> 1.756617036538e-07 ||r(i)||/||b|| 2.384847135242e-08
> > > > 10 KSP preconditioned resid norm 2.480228743414e-11 true resid norm
> 1.309399823577e-08 ||r(i)||/||b|| 1.777688792258e-09
> > > > 11 KSP preconditioned resid norm 1.728967759950e-12 true resid norm
> 9.406967045789e-10 ||r(i)||/||b|| 1.277124036931e-10
> > > > 12 KSP preconditioned resid norm 1.075458632828e-13 true resid norm
> 5.994505136212e-11 ||r(i)||/||b|| 8.138358050689e-12
> > > > Linear solve converged due to CONVERGED_RTOL iterations 12
> > > > KSP Object: 24 MPI processes
> > > > type: gmres
> > > > restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > > > happy breakdown tolerance 1e-30
> > > > maximum iterations=40000, initial guess is zero
> > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > left preconditioning
> > > > using PRECONDITIONED norm type for convergence test
> > > > PC Object: 24 MPI processes
> > > > type: hypre
> > > > HYPRE BoomerAMG preconditioning
> > > > Cycle type V
> > > > Maximum number of levels 25
> > > > Maximum number of iterations PER hypre call 1
> > > > Convergence tolerance PER hypre call 0.
> > > > Threshold for strong coupling 0.25
> > > > Interpolation truncation factor 0.
> > > > Interpolation: max elements per row 0
> > > > Number of levels of aggressive coarsening 0
> > > > Number of paths for aggressive coarsening 1
> > > > Maximum row sums 0.9
> > > > Sweeps down 1
> > > > Sweeps up 1
> > > > Sweeps on coarse 1
> > > > Relax down symmetric-SOR/Jacobi
> > > > Relax up symmetric-SOR/Jacobi
> > > > Relax on coarse Gaussian-elimination
> > > > Relax weight (all) 1.
> > > > Outer relax weight (all) 1.
> > > > Using CF-relaxation
> > > > Not using more complex smoothers.
> > > > Measure type local
> > > > Coarsen type Falgout
> > > > Interpolation type classical
> > > > linear system matrix = precond matrix:
> > > > Mat Object: A 24 MPI processes
> > > > type: mpiaij
> > > > rows=497664, cols=497664
> > > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > > total number of mallocs used during MatSetValues calls =0
> > > > has attached null space
> > > > not using I-node (on process 0) routines
> > > > The max singular value of A = 1.000872 in poisson_solver3d_P0_vd
> > > > The min singular value of A = 0.667688 in poisson_solver3d_P0_vd
> > > > The Cond Num of A = 1.499012 in poisson_solver3d_P0_vd
> > > > In poisson_solver3d_pressure(): num_iter = 12, rel_residual =
> 1.075459e-13
> > > >
> > > > The max value of p0 is 0.03115845493408858
> > > >
> > > > The min value of p0 is -0.07156715468428149
> > > >
> > > > On Sun, Oct 22, 2017 at 12:00 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > >
> > > > > On Oct 21, 2017, at 10:50 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > >
> > > > > the incompressible NS solver algorithm call PETSc solver at
> different stage of each time step. The one you were saying "This is good.
> 12 digit reduction" is after the initial pressure solver, in which usually
> HYPRE doesn't give a good convergence, so the fall-back solver GMRES will
> be called after.
> > > >
> > > > Hmm, I don't understand. hypre should do well on a pressure solve.
> In fact, very well.
> > > > >
> > > > > Barry, you were mentioning that I could have a wrong nullspace.
> that particular solver is aimed to give an initial pressure profile for 3d
> incompressible NS simulation using all neumann boundary conditions. could
> you give some insight how to test if I have a wrong nullspace etc?
> > > >
> > > > -ksp_test_null_space
> > > >
> > > > But if your null space is consistently from all Neumann boundary
> conditions then it likely is not wrong.
> > > >
> > > > Barry
> > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Sat, Oct 21, 2017 at 11:41 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > >
> > > > > This is good. You get more than 12 digit reduction in the true
> residual norm. This is good AMG convergence. Expected when everything goes
> well.
> > > > >
> > > > > What is different in this case from the previous case that does
> not converge reasonably?
> > > > >
> > > > > Barry
> > > > >
> > > > >
> > > > > > On Oct 21, 2017, at 9:29 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > > >
> > > > > > Barry, Please advise what you make of this? this is poisson
> solver with all neumann BC 3d case Finite difference Scheme was used.
> > > > > > Thanks! I'm in learning mode.
> > > > > >
> > > > > > KSP Object: 24 MPI processes
> > > > > > type: bcgs
> > > > > > maximum iterations=40000, initial guess is zero
> > > > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > > > left preconditioning
> > > > > > using PRECONDITIONED norm type for convergence test
> > > > > > PC Object: 24 MPI processes
> > > > > > type: hypre
> > > > > > HYPRE BoomerAMG preconditioning
> > > > > > Cycle type V
> > > > > > Maximum number of levels 25
> > > > > > Maximum number of iterations PER hypre call 1
> > > > > > Convergence tolerance PER hypre call 0.
> > > > > > Threshold for strong coupling 0.25
> > > > > > Interpolation truncation factor 0.
> > > > > > Interpolation: max elements per row 0
> > > > > > Number of levels of aggressive coarsening 0
> > > > > > Number of paths for aggressive coarsening 1
> > > > > > Maximum row sums 0.9
> > > > > > Sweeps down 1
> > > > > > Sweeps up 1
> > > > > > Sweeps on coarse 1
> > > > > > Relax down symmetric-SOR/Jacobi
> > > > > > Relax up symmetric-SOR/Jacobi
> > > > > > Relax on coarse Gaussian-elimination
> > > > > > Relax weight (all) 1.
> > > > > > Outer relax weight (all) 1.
> > > > > > Using CF-relaxation
> > > > > > Not using more complex smoothers.
> > > > > > Measure type local
> > > > > > Coarsen type Falgout
> > > > > > Interpolation type classical
> > > > > > linear system matrix = precond matrix:
> > > > > > Mat Object: A 24 MPI processes
> > > > > > type: mpiaij
> > > > > > rows=497664, cols=497664
> > > > > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > > > > total number of mallocs used during MatSetValues calls =0
> > > > > > has attached null space
> > > > > > not using I-node (on process 0) routines
> > > > > > 0 KSP preconditioned resid norm 2.697270170623e-02 true resid
> norm 9.637159071207e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > > > > 1 KSP preconditioned resid norm 4.828857674609e-04 true resid
> norm 6.294379664645e-01 ||r(i)||/||b|| 6.531364293291e-02
> > > > > > 2 KSP preconditioned resid norm 4.533649595815e-06 true resid
> norm 1.135508605857e-02 ||r(i)||/||b|| 1.178260727531e-03
> > > > > > 3 KSP preconditioned resid norm 1.131704082606e-07 true resid
> norm 1.196393029874e-04 ||r(i)||/||b|| 1.241437462051e-05
> > > > > > 4 KSP preconditioned resid norm 3.866281379569e-10 true resid
> norm 5.121520801846e-07 ||r(i)||/||b|| 5.314347064320e-08
> > > > > > 5 KSP preconditioned resid norm 1.070114785241e-11 true resid
> norm 1.203693733135e-08 ||r(i)||/||b|| 1.249013038221e-09
> > > > > > 6 KSP preconditioned resid norm 2.578780418765e-14 true resid
> norm 6.020297525927e-11 ||r(i)||/||b|| 6.246962908306e-12
> > > > > > 7 KSP preconditioned resid norm 8.691764190203e-16 true resid
> norm 1.866088098154e-12 ||r(i)||/||b|| 1.936346680973e-13
> > > > > > Linear solve converged due to CONVERGED_ATOL iterations 7
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Oct 21, 2017 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > > >
> > > > > > > On Oct 21, 2017, at 5:47 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > > > >
> > > > > > > hi, Barry:
> > > > > > > what do you mean absurd by setting tolerance =1e-14?
> > > > > >
> > > > > > Trying to decrease the initial residual norm down by a factor
> of 1e-14 with an iterative method (or even direct method) is unrealistic,
> usually unachievable) and almost never necessary. You are requiring || r_n
> || < 1.e-14 || r_0|| when with double precision numbers you only have
> roughly 14 decimal digits total to compute with. Round off alone will lead
> to differences far larger than 1e-14
> > > > > >
> > > > > > If you are using the solver in the context of a nonlinear
> problem (i.e. inside Newton's method) then 1.e-6 is generally more than
> plenty to get quadratic convergence of Newton's method.
> > > > > >
> > > > > > If you are solving a linear problem then it is extremely
> likely that errors due to discretization errors (from finite element method
> etc) and the model are much much larger than even 1.e-8.
> > > > > >
> > > > > > So, in summary
> > > > > >
> > > > > > 1.e-14 is probably unachievable
> > > > > >
> > > > > > 1.e-14 is almost for sure not needed.
> > > > > >
> > > > > > Barry
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > On Sat, Oct 21, 2017 at 18:42 Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > > > >
> > > > > > > Run with -ksp_view_mat binary -ksp_view_rhs binary and send
> the resulting output file called binaryoutput to petsc-maint at mcs.anl.gov
> > > > > > >
> > > > > > > Note you can also use -ksp_type gmres with hypre, unlikely
> to be a reason to use bcgs
> > > > > > >
> > > > > > > BTW: tolerances: relative=1e-14, is absurd
> > > > > > >
> > > > > > > My guess is your null space is incorrect.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > On Oct 21, 2017, at 4:34 PM, Hao Zhang <hbcbh1999 at gmail.com>
> wrote:
> > > > > > > >
> > > > > > > > if this solver doesn't converge. I have a fall-back
> solution, which uses GMRES solver. this setup is fine with me. I just want
> to know if HYPRE is a reliable solution for me. Or I will have to go
> without preconditioner.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > On Sat, Oct 21, 2017 at 5:30 PM, Hao Zhang <
> hbcbh1999 at gmail.com> wrote:
> > > > > > > > this is serial run. still dumping output. parallel more or
> less the same.
> > > > > > > >
> > > > > > > > KSP Object: 1 MPI processes
> > > > > > > > type: bcgs
> > > > > > > > maximum iterations=40000, initial guess is zero
> > > > > > > > tolerances: relative=1e-14, absolute=1e-14,
> divergence=10000.
> > > > > > > > left preconditioning
> > > > > > > > using PRECONDITIONED norm type for convergence test
> > > > > > > > PC Object: 1 MPI processes
> > > > > > > > type: hypre
> > > > > > > > HYPRE BoomerAMG preconditioning
> > > > > > > > Cycle type V
> > > > > > > > Maximum number of levels 25
> > > > > > > > Maximum number of iterations PER hypre call 1
> > > > > > > > Convergence tolerance PER hypre call 0.
> > > > > > > > Threshold for strong coupling 0.25
> > > > > > > > Interpolation truncation factor 0.
> > > > > > > > Interpolation: max elements per row 0
> > > > > > > > Number of levels of aggressive coarsening 0
> > > > > > > > Number of paths for aggressive coarsening 1
> > > > > > > > Maximum row sums 0.9
> > > > > > > > Sweeps down 1
> > > > > > > > Sweeps up 1
> > > > > > > > Sweeps on coarse 1
> > > > > > > > Relax down symmetric-SOR/Jacobi
> > > > > > > > Relax up symmetric-SOR/Jacobi
> > > > > > > > Relax on coarse Gaussian-elimination
> > > > > > > > Relax weight (all) 1.
> > > > > > > > Outer relax weight (all) 1.
> > > > > > > > Using CF-relaxation
> > > > > > > > Not using more complex smoothers.
> > > > > > > > Measure type local
> > > > > > > > Coarsen type Falgout
> > > > > > > > Interpolation type classical
> > > > > > > > linear system matrix = precond matrix:
> > > > > > > > Mat Object: A 1 MPI processes
> > > > > > > > type: seqaij
> > > > > > > > rows=497664, cols=497664
> > > > > > > > total: nonzeros=3363552, allocated nonzeros=3483648
> > > > > > > > total number of mallocs used during MatSetValues calls =0
> > > > > > > > has attached null space
> > > > > > > > not using I-node routines
> > > > > > > > 0 KSP preconditioned resid norm 1.630377897956e+01 true
> resid norm 7.365742695123e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > > > > > > 1 KSP preconditioned resid norm 3.815819909205e-02 true
> resid norm 7.592300567353e-01 ||r(i)||/||b|| 1.030758320187e-01
> > > > > > > > 2 KSP preconditioned resid norm 4.312671277701e-04 true
> resid norm 2.553060965521e-02 ||r(i)||/||b|| 3.466128360975e-03
> > > > > > > > 3 KSP preconditioned resid norm 3.011875330569e-04 true
> resid norm 6.836208627386e-04 ||r(i)||/||b|| 9.281085303065e-05
> > > > > > > > 4 KSP preconditioned resid norm 3.011783821295e-04 true
> resid norm 2.504661140204e-04 ||r(i)||/||b|| 3.400418998972e-05
> > > > > > > > 5 KSP preconditioned resid norm 3.011783818372e-04 true
> resid norm 2.504053673004e-04 ||r(i)||/||b|| 3.399594279422e-05
> > > > > > > > 6 KSP preconditioned resid norm 3.011783818375e-04 true
> resid norm 2.503984078890e-04 ||r(i)||/||b|| 3.399499795925e-05
> > > > > > > > 7 KSP preconditioned resid norm 3.011783887442e-04 true
> resid norm 2.504121501772e-04 ||r(i)||/||b|| 3.399686366224e-05
> > > > > > > > 8 KSP preconditioned resid norm 3.010913654181e-04 true
> resid norm 2.504150259031e-04 ||r(i)||/||b|| 3.399725408124e-05
> > > > > > > > 9 KSP preconditioned resid norm 3.006520688232e-04 true
> resid norm 2.504061607382e-04 ||r(i)||/||b|| 3.399605051423e-05
> > > > > > > > 10 KSP preconditioned resid norm 3.007309991942e-04 true
> resid norm 2.503843638523e-04 ||r(i)||/||b|| 3.399309128978e-05
> > > > > > > > 11 KSP preconditioned resid norm 3.015946168077e-04 true
> resid norm 2.503644677844e-04 ||r(i)||/||b|| 3.399039012728e-05
> > > > > > > > 12 KSP preconditioned resid norm 2.956643907377e-04 true
> resid norm 2.503863046509e-04 ||r(i)||/||b|| 3.399335477965e-05
> > > > > > > > 13 KSP preconditioned resid norm 2.997992358936e-04 true
> resid norm 2.504336903093e-04 ||r(i)||/||b|| 3.399978802886e-05
> > > > > > > > 14 KSP preconditioned resid norm 2.481415420420e-05 true
> resid norm 2.491591201250e-04 ||r(i)||/||b|| 3.382674774806e-05
> > > > > > > > 15 KSP preconditioned resid norm 2.615494786181e-05 true
> resid norm 2.490353237273e-04 ||r(i)||/||b|| 3.380994069915e-05
> > > > > > > > 16 KSP preconditioned resid norm 2.645126692130e-05 true
> resid norm 2.490535523344e-04 ||r(i)||/||b|| 3.381241548111e-05
> > > > > > > > 17 KSP preconditioned resid norm 2.667223026209e-05 true
> resid norm 2.490482602536e-04 ||r(i)||/||b|| 3.381169700898e-05
> > > > > > > > 18 KSP preconditioned resid norm 2.650813432116e-05 true
> resid norm 2.490473169014e-04 ||r(i)||/||b|| 3.381156893606e-05
> > > > > > > > 19 KSP preconditioned resid norm 2.613309555449e-05 true
> resid norm 2.490465633690e-04 ||r(i)||/||b|| 3.381146663375e-05
> > > > > > > > 20 KSP preconditioned resid norm 2.644160446804e-05 true
> resid norm 2.490532739949e-04 ||r(i)||/||b|| 3.381237769272e-05
> > > > > > > > 21 KSP preconditioned resid norm 2.635987608975e-05 true
> resid norm 2.490499548926e-04 ||r(i)||/||b|| 3.381192707933e-05
> > > > > > > > 22 KSP preconditioned resid norm 2.640527129095e-05 true
> resid norm 2.490594066529e-04 ||r(i)||/||b|| 3.381321028466e-05
> > > > > > > > 23 KSP preconditioned resid norm 2.627505117691e-05 true
> resid norm 2.490550162585e-04 ||r(i)||/||b|| 3.381261422875e-05
> > > > > > > > 24 KSP preconditioned resid norm 2.642659196388e-05 true
> resid norm 2.490504347640e-04 ||r(i)||/||b|| 3.381199222842e-05
> > > > > > > > 25 KSP preconditioned resid norm 2.659432190695e-05 true
> resid norm 2.490510775152e-04 ||r(i)||/||b|| 3.381207949065e-05
> > > > > > > > 26 KSP preconditioned resid norm 2.687918062951e-05 true
> resid norm 2.490518882015e-04 ||r(i)||/||b|| 3.381218955237e-05
> > > > > > > > 27 KSP preconditioned resid norm 2.662909048432e-05 true
> resid norm 2.490446263285e-04 ||r(i)||/||b|| 3.381120365409e-05
> > > > > > > > 28 KSP preconditioned resid norm 2.085466483199e-05 true
> resid norm 2.490131612366e-04 ||r(i)||/||b|| 3.380693183886e-05
> > > > > > > > 29 KSP preconditioned resid norm 2.098541330282e-05 true
> resid norm 2.490126933398e-04 ||r(i)||/||b|| 3.380686831549e-05
> > > > > > > > 30 KSP preconditioned resid norm 2.175345180286e-05 true
> resid norm 2.490098852429e-04 ||r(i)||/||b|| 3.380648707805e-05
> > > > > > > > 31 KSP preconditioned resid norm 2.182182437676e-05 true
> resid norm 2.490028301020e-04 ||r(i)||/||b|| 3.380552924648e-05
> > > > > > > > 32 KSP preconditioned resid norm 2.152970404369e-05 true
> resid norm 2.490089939838e-04 ||r(i)||/||b|| 3.380636607747e-05
> > > > > > > > 33 KSP preconditioned resid norm 2.187932450016e-05 true
> resid norm 2.490085293931e-04 ||r(i)||/||b|| 3.380630300295e-05
> > > > > > > > 34 KSP preconditioned resid norm 2.207255875067e-05 true
> resid norm 2.490039036092e-04 ||r(i)||/||b|| 3.380567498971e-05
> > > > > > > > 35 KSP preconditioned resid norm 2.205060279701e-05 true
> resid norm 2.490101636150e-04 ||r(i)||/||b|| 3.380652487086e-05
> > > > > > > > 36 KSP preconditioned resid norm 2.168654200416e-05 true
> resid norm 2.490091609876e-04 ||r(i)||/||b|| 3.380638875052e-05
> > > > > > > > 37 KSP preconditioned resid norm 2.164521042361e-05 true
> resid norm 2.490083143913e-04 ||r(i)||/||b|| 3.380627381352e-05
> > > > > > > > 38 KSP preconditioned resid norm 2.154429063973e-05 true
> resid norm 2.490075485470e-04 ||r(i)||/||b|| 3.380616983972e-05
> > > > > > > > 39 KSP preconditioned resid norm 2.165962086228e-05 true
> resid norm 2.490099695056e-04 ||r(i)||/||b|| 3.380649851786e-05
> > > > > > > > 40 KSP preconditioned resid norm 2.153877616091e-05 true
> resid norm 2.490090652619e-04 ||r(i)||/||b|| 3.380637575444e-05
> > > > > > > > 41 KSP preconditioned resid norm 2.347651187611e-05 true
> resid norm 2.490233544624e-04 ||r(i)||/||b|| 3.380831570825e-05
> > > > > > > > 42 KSP preconditioned resid norm 2.352860162514e-05 true
> resid norm 2.490191394202e-04 ||r(i)||/||b|| 3.380774345879e-05
> > > > > > > > 43 KSP preconditioned resid norm 2.312377506928e-05 true
> resid norm 2.490209491359e-04 ||r(i)||/||b|| 3.380798915237e-05
> > > > > > > > 44 KSP preconditioned resid norm 2.295770973533e-05 true
> resid norm 2.490178136759e-04 ||r(i)||/||b|| 3.380756347093e-05
> > > > > > > > 45 KSP preconditioned resid norm 2.833646456041e-05 true
> resid norm 2.489991602651e-04 ||r(i)||/||b|| 3.380503101608e-05
> > > > > > > > 46 KSP preconditioned resid norm 2.760296424494e-05 true
> resid norm 2.490104320666e-04 ||r(i)||/||b|| 3.380656131682e-05
> > > > > > > > 47 KSP preconditioned resid norm 2.451504295239e-05 true
> resid norm 2.490241388672e-04 ||r(i)||/||b|| 3.380842220189e-05
> > > > > > > > 48 KSP preconditioned resid norm 2.512391514098e-05 true
> resid norm 2.490245923753e-04 ||r(i)||/||b|| 3.380848377180e-05
> > > > > > > > 49 KSP preconditioned resid norm 2.483419450528e-05 true
> resid norm 2.490273364402e-04 ||r(i)||/||b|| 3.380885631602e-05
> > > > > > > > 50 KSP preconditioned resid norm 2.507460538466e-05 true
> resid norm 2.490309488780e-04 ||r(i)||/||b|| 3.380934675371e-05
> > > > > > > > 51 KSP preconditioned resid norm 2.499708772881e-05 true
> resid norm 2.490300908170e-04 ||r(i)||/||b|| 3.380923026022e-05
> > > > > > > > 52 KSP preconditioned resid norm 1.059778259446e-05 true
> resid norm 2.489352833521e-04 ||r(i)||/||b|| 3.379635885420e-05
> > > > > > > > 53 KSP preconditioned resid norm 1.074975117060e-05 true
> resid norm 2.489294722901e-04 ||r(i)||/||b|| 3.379556992330e-05
> > > > > > > > 54 KSP preconditioned resid norm 1.095242219559e-05 true
> resid norm 2.489295454212e-04 ||r(i)||/||b|| 3.379557985184e-05
> > > > > > > > 55 KSP preconditioned resid norm 8.359999674720e-06 true
> resid norm 2.489673581944e-04 ||r(i)||/||b|| 3.380071345137e-05
> > > > > > > > 56 KSP preconditioned resid norm 8.368232998470e-06 true
> resid norm 2.489700421343e-04 ||r(i)||/||b|| 3.380107783281e-05
> > > > > > > > 57 KSP preconditioned resid norm 8.443378041101e-06 true
> resid norm 2.489702900875e-04 ||r(i)||/||b|| 3.380111149584e-05
> > > > > > > > 58 KSP preconditioned resid norm 8.647159584302e-06 true
> resid norm 2.489640805831e-04 ||r(i)||/||b|| 3.380026847095e-05
> > > > > > > > 59 KSP preconditioned resid norm 1.024742790737e-05 true
> resid norm 2.489447846660e-04 ||r(i)||/||b|| 3.379764878711e-05
> > > > > > > > 60 KSP preconditioned resid norm 1.033394118910e-05 true
> resid norm 2.489441404923e-04 ||r(i)||/||b|| 3.379756133175e-05
> > > > > > > > 61 KSP preconditioned resid norm 1.030066336446e-05 true
> resid norm 2.489399918556e-04 ||r(i)||/||b|| 3.379699809776e-05
> > > > > > > > 62 KSP preconditioned resid norm 1.029956398963e-05 true
> resid norm 2.489445295139e-04 ||r(i)||/||b|| 3.379761414674e-05
> > > > > > > > 63 KSP preconditioned resid norm 1.028190129002e-05 true
> resid norm 2.489456200527e-04 ||r(i)||/||b|| 3.379776220225e-05
> > > > > > > > 64 KSP preconditioned resid norm 9.878799185773e-06 true
> resid norm 2.489488742330e-04 ||r(i)||/||b|| 3.379820400160e-05
> > > > > > > > 65 KSP preconditioned resid norm 9.917711104174e-06 true
> resid norm 2.489478066593e-04 ||r(i)||/||b|| 3.379805906391e-05
> > > > > > > > 66 KSP preconditioned resid norm 1.003572019576e-05 true
> resid norm 2.489441995703e-04 ||r(i)||/||b|| 3.379756935240e-05
> > > > > > > > 67 KSP preconditioned resid norm 9.924487278236e-06 true
> resid norm 2.489475403451e-04 ||r(i)||/||b|| 3.379802290812e-05
> > > > > > > > 68 KSP preconditioned resid norm 9.804213483359e-06 true
> resid norm 2.489457781760e-04 ||r(i)||/||b|| 3.379778366964e-05
> > > > > > > > 69 KSP preconditioned resid norm 9.748922705476e-06 true
> resid norm 2.489408473578e-04 ||r(i)||/||b|| 3.379711424383e-05
> > > > > > > > 70 KSP preconditioned resid norm 9.886044523689e-06 true
> resid norm 2.489514438395e-04 ||r(i)||/||b|| 3.379855286071e-05
> > > > > > > > 71 KSP preconditioned resid norm 1.083888478689e-05 true
> resid norm 2.489420898851e-04 ||r(i)||/||b|| 3.379728293386e-05
> > > > > > > > 72 KSP preconditioned resid norm 1.106561823757e-05 true
> resid norm 2.489364778104e-04 ||r(i)||/||b|| 3.379652101821e-05
> > > > > > > > 73 KSP preconditioned resid norm 1.132091515426e-05 true
> resid norm 2.489456804535e-04 ||r(i)||/||b|| 3.379777040248e-05
> > > > > > > > 74 KSP preconditioned resid norm 1.330905328963e-05 true
> resid norm 2.489317458981e-04 ||r(i)||/||b|| 3.379587859660e-05
> > > > > > > > 75 KSP preconditioned resid norm 1.305555302619e-05 true
> resid norm 2.489320939810e-04 ||r(i)||/||b|| 3.379592585359e-05
> > > > > > > > 76 KSP preconditioned resid norm 1.308083397399e-05 true
> resid norm 2.489299951581e-04 ||r(i)||/||b|| 3.379564090977e-05
> > > > > > > > 77 KSP preconditioned resid norm 1.320098861853e-05 true
> resid norm 2.489323669317e-04 ||r(i)||/||b|| 3.379596291036e-05
> > > > > > > > 78 KSP preconditioned resid norm 1.300160788274e-05 true
> resid norm 2.489306393356e-04 ||r(i)||/||b|| 3.379572836564e-05
> > > > > > > > 79 KSP preconditioned resid norm 1.317651537793e-05 true
> resid norm 2.489381364970e-04 ||r(i)||/||b|| 3.379674620752e-05
> > > > > > > > 80 KSP preconditioned resid norm 1.309769805765e-05 true
> resid norm 2.489285056062e-04 ||r(i)||/||b|| 3.379543868279e-05
> > > > > > > > 81 KSP preconditioned resid norm 1.293686496271e-05 true
> resid norm 2.489347818072e-04 ||r(i)||/||b|| 3.379629076264e-05
> > > > > > > > 82 KSP preconditioned resid norm 1.311788285799e-05 true
> resid norm 2.489320040215e-04 ||r(i)||/||b|| 3.379591364037e-05
> > > > > > > > 83 KSP preconditioned resid norm 1.313667378798e-05 true
> resid norm 2.489329437217e-04 ||r(i)||/||b|| 3.379604121748e-05
> > > > > > > > 84 KSP preconditioned resid norm 1.416138205017e-05 true
> resid norm 2.489266908838e-04 ||r(i)||/||b|| 3.379519230948e-05
> > > > > > > > 85 KSP preconditioned resid norm 1.452253464774e-05 true
> resid norm 2.489285688375e-04 ||r(i)||/||b|| 3.379544726729e-05
> > > > > > > > 86 KSP preconditioned resid norm 1.426709413370e-05 true
> resid norm 2.489362313402e-04 ||r(i)||/||b|| 3.379648755651e-05
> > > > > > > > 87 KSP preconditioned resid norm 1.427480849552e-05 true
> resid norm 2.489378183000e-04 ||r(i)||/||b|| 3.379670300795e-05
> > > > > > > > 88 KSP preconditioned resid norm 1.413870980147e-05 true
> resid norm 2.489325756118e-04 ||r(i)||/||b|| 3.379599124153e-05
> > > > > > > > 89 KSP preconditioned resid norm 1.353259857657e-05 true
> resid norm 2.489318968308e-04 ||r(i)||/||b|| 3.379589908776e-05
> > > > > > > > 90 KSP preconditioned resid norm 1.347676448611e-05 true
> resid norm 2.489332074417e-04 ||r(i)||/||b|| 3.379607702106e-05
> > > > > > > > 91 KSP preconditioned resid norm 1.362825902909e-05 true
> resid norm 2.489344974971e-04 ||r(i)||/||b|| 3.379625216367e-05
> > > > > > > > 92 KSP preconditioned resid norm 1.346280901052e-05 true
> resid norm 2.489302570131e-04 ||r(i)||/||b|| 3.379567646016e-05
> > > > > > > > 93 KSP preconditioned resid norm 1.328052169696e-05 true
> resid norm 2.489346601224e-04 ||r(i)||/||b|| 3.379627424228e-05
> > > > > > > > 94 KSP preconditioned resid norm 1.554682082515e-05 true
> resid norm 2.489309078759e-04 ||r(i)||/||b|| 3.379576482365e-05
> > > > > > > > 95 KSP preconditioned resid norm 1.557128675775e-05 true
> resid norm 2.489317143582e-04 ||r(i)||/||b|| 3.379587431462e-05
> > > > > > > > 96 KSP preconditioned resid norm 1.542571813923e-05 true
> resid norm 2.489319910303e-04 ||r(i)||/||b|| 3.379591187663e-05
> > > > > > > > 97 KSP preconditioned resid norm 1.570516684444e-05 true
> resid norm 2.489321980894e-04 ||r(i)||/||b|| 3.379593998772e-05
> > > > > > > > 98 KSP preconditioned resid norm 1.600431789899e-05 true
> resid norm 2.489297450311e-04 ||r(i)||/||b|| 3.379560695162e-05
> > > > > > > > 99 KSP preconditioned resid norm 1.587495554658e-05 true
> resid norm 2.489339000570e-04 ||r(i)||/||b|| 3.379617105303e-05
> > > > > > > > 100 KSP preconditioned resid norm 1.621163002878e-05 true
> resid norm 2.489299953360e-04 ||r(i)||/||b|| 3.379564093392e-05
> > > > > > > > 101 KSP preconditioned resid norm 1.627060872574e-05 true
> resid norm 2.489301570161e-04 ||r(i)||/||b|| 3.379566288419e-05
> > > > > > > > 102 KSP preconditioned resid norm 1.622931647243e-05 true
> resid norm 2.489277930910e-04 ||r(i)||/||b|| 3.379534194913e-05
> > > > > > > > 103 KSP preconditioned resid norm 1.612544300282e-05 true
> resid norm 2.489317483299e-04 ||r(i)||/||b|| 3.379587892674e-05
> > > > > > > > 104 KSP preconditioned resid norm 1.880131646630e-05 true
> resid norm 2.489335862583e-04 ||r(i)||/||b|| 3.379612845059e-05
> > > > > > > > 105 KSP preconditioned resid norm 1.880563295793e-05 true
> resid norm 2.489365017923e-04 ||r(i)||/||b|| 3.379652427408e-05
> > > > > > > > 106 KSP preconditioned resid norm 1.860619184027e-05 true
> resid norm 2.489362382373e-04 ||r(i)||/||b|| 3.379648849288e-05
> > > > > > > > 107 KSP preconditioned resid norm 1.877134148719e-05 true
> resid norm 2.489425484523e-04 ||r(i)||/||b|| 3.379734519061e-05
> > > > > > > > 108 KSP preconditioned resid norm 1.914810713538e-05 true
> resid norm 2.489347415573e-04 ||r(i)||/||b|| 3.379628529818e-05
> > > > > > > > 109 KSP preconditioned resid norm 1.220673255622e-05 true
> resid norm 2.490196186357e-04 ||r(i)||/||b|| 3.380780851884e-05
> > > > > > > > 110 KSP preconditioned resid norm 1.215819132910e-05 true
> resid norm 2.490233518072e-04 ||r(i)||/||b|| 3.380831534776e-05
> > > > > > > > 111 KSP preconditioned resid norm 1.196565427400e-05 true
> resid norm 2.490279438906e-04 ||r(i)||/||b|| 3.380893878570e-05
> > > > > > > > 112 KSP preconditioned resid norm 1.171748185197e-05 true
> resid norm 2.490242578983e-04 ||r(i)||/||b|| 3.380843836198e-05
> > > > > > > > 113 KSP preconditioned resid norm 1.162855824118e-05 true
> resid norm 2.490229018536e-04 ||r(i)||/||b|| 3.380825426043e-05
> > > > > > > > 114 KSP preconditioned resid norm 1.175594685689e-05 true
> resid norm 2.490274328440e-04 ||r(i)||/||b|| 3.380886940415e-05
> > > > > > > > 115 KSP preconditioned resid norm 1.167979454122e-05 true
> resid norm 2.490271161036e-04 ||r(i)||/||b|| 3.380882640232e-05
> > > > > > > > 116 KSP preconditioned resid norm 1.181010893019e-05 true
> resid norm 2.490235420657e-04 ||r(i)||/||b|| 3.380834117795e-05
> > > > > > > > 117 KSP preconditioned resid norm 1.175206638194e-05 true
> resid norm 2.490263165345e-04 ||r(i)||/||b|| 3.380871784992e-05
> > > > > > > > 118 KSP preconditioned resid norm 1.183804125791e-05 true
> resid norm 2.490221353083e-04 ||r(i)||/||b|| 3.380815019145e-05
> > > > > > > > 119 KSP preconditioned resid norm 1.186426973727e-05 true
> resid norm 2.490227115336e-04 ||r(i)||/||b|| 3.380822842189e-05
> > > > > > > > 120 KSP preconditioned resid norm 1.181986776689e-05 true
> resid norm 2.490257884230e-04 ||r(i)||/||b|| 3.380864615159e-05
> > > > > > > > 121 KSP preconditioned resid norm 1.131443277370e-05 true
> resid norm 2.490259110230e-04 ||r(i)||/||b|| 3.380866279620e-05
> > > > > > > > 122 KSP preconditioned resid norm 1.114920075859e-05 true
> resid norm 2.490249829382e-04 ||r(i)||/||b|| 3.380853679603e-05
> > > > > > > > 123 KSP preconditioned resid norm 1.082073321672e-05 true
> resid norm 2.490314084868e-04 ||r(i)||/||b|| 3.380940915187e-05
> > > > > > > > 124 KSP preconditioned resid norm 3.307785860990e-06 true
> resid norm 2.490613501549e-04 ||r(i)||/||b|| 3.381347414155e-05
> > > > > > > > 125 KSP preconditioned resid norm 3.287051720572e-06 true
> resid norm 2.490584648195e-04 ||r(i)||/||b|| 3.381308241794e-05
> > > > > > > > 126 KSP preconditioned resid norm 3.286797046069e-06 true
> resid norm 2.490654396386e-04 ||r(i)||/||b|| 3.381402934473e-05
> > > > > > > > 127 KSP preconditioned resid norm 3.311592899411e-06 true
> resid norm 2.490627973588e-04 ||r(i)||/||b|| 3.381367061922e-05
> > > > > > > > 128 KSP preconditioned resid norm 3.560993694635e-06 true
> resid norm 2.490571732816e-04 ||r(i)||/||b|| 3.381290707406e-05
> > > > > > > > 129 KSP preconditioned resid norm 3.411994617661e-06 true
> resid norm 2.490652122141e-04 ||r(i)||/||b|| 3.381399846875e-05
> > > > > > > > 130 KSP preconditioned resid norm 3.412383310721e-06 true
> resid norm 2.490633454022e-04 ||r(i)||/||b|| 3.381374502359e-05
> > > > > > > > 131 KSP preconditioned resid norm 3.288320044878e-06 true
> resid norm 2.490639470096e-04 ||r(i)||/||b|| 3.381382669999e-05
> > > > > > > > 132 KSP preconditioned resid norm 3.273215756565e-06 true
> resid norm 2.490640390847e-04 ||r(i)||/||b|| 3.381383920043e-05
> > > > > > > > 133 KSP preconditioned resid norm 3.236969051459e-06 true
> resid norm 2.490678216102e-04 ||r(i)||/||b|| 3.381435272985e-05
> > > > > > > > 134 KSP preconditioned resid norm 3.203260913942e-06 true
> resid norm 2.490640965346e-04 ||r(i)||/||b|| 3.381384700005e-05
> > > > > > > > 135 KSP preconditioned resid norm 3.224117152353e-06 true
> resid norm 2.490655026376e-04 ||r(i)||/||b|| 3.381403789770e-05
> > > > > > > > 136 KSP preconditioned resid norm 3.221577997984e-06 true
> resid norm 2.490684737611e-04 ||r(i)||/||b|| 3.381444126823e-05
> > > > > > > > 137 KSP preconditioned resid norm 3.195936222128e-06 true
> resid norm 2.490673982333e-04 ||r(i)||/||b|| 3.381429525066e-05
> > > > > > > > 138 KSP preconditioned resid norm 3.207528137426e-06 true
> resid norm 2.490641247196e-04 ||r(i)||/||b|| 3.381385082655e-05
> > > > > > > > 139 KSP preconditioned resid norm 3.240134271963e-06 true
> resid norm 2.490615861251e-04 ||r(i)||/||b|| 3.381350617773e-05
> > > > > > > > 140 KSP preconditioned resid norm 2.698833607230e-06 true
> resid norm 2.490638954889e-04 ||r(i)||/||b|| 3.381381970535e-05
> > > > > > > > 141 KSP preconditioned resid norm 2.599151209137e-06 true
> resid norm 2.490657106698e-04 ||r(i)||/||b|| 3.381406614091e-05
> > > > > > > > 142 KSP preconditioned resid norm 2.633939920994e-06 true
> resid norm 2.490707754695e-04 ||r(i)||/||b|| 3.381475375652e-05
> > > > > > > > 143 KSP preconditioned resid norm 2.519609221376e-06 true
> resid norm 2.490639100480e-04 ||r(i)||/||b|| 3.381382168195e-05
> > > > > > > > 144 KSP preconditioned resid norm 3.768526937684e-06 true
> resid norm 2.490654096698e-04 ||r(i)||/||b|| 3.381402527606e-05
> > > > > > > > 145 KSP preconditioned resid norm 3.707841943289e-06 true
> resid norm 2.490630207923e-04 ||r(i)||/||b|| 3.381370095336e-05
> > > > > > > > 146 KSP preconditioned resid norm 3.698827503486e-06 true
> resid norm 2.490646071561e-04 ||r(i)||/||b|| 3.381391632387e-05
> > > > > > > > 147 KSP preconditioned resid norm 3.642747039615e-06 true
> resid norm 2.490610990161e-04 ||r(i)||/||b|| 3.381344004604e-05
> > > > > > > > 148 KSP preconditioned resid norm 3.613100087842e-06 true
> resid norm 2.490617159023e-04 ||r(i)||/||b|| 3.381352379676e-05
> > > > > > > > 149 KSP preconditioned resid norm 3.637646399299e-06 true
> resid norm 2.490648063023e-04 ||r(i)||/||b|| 3.381394336069e-05
> > > > > > > > 150 KSP preconditioned resid norm 3.640235367864e-06 true
> resid norm 2.490648516718e-04 ||r(i)||/||b|| 3.381394952022e-05
> > > > > > > > 151 KSP preconditioned resid norm 3.724708848977e-06 true
> resid norm 2.490622201040e-04 ||r(i)||/||b|| 3.381359224901e-05
> > > > > > > > 152 KSP preconditioned resid norm 3.665185002770e-06 true
> resid norm 2.490664302790e-04 ||r(i)||/||b|| 3.381416383766e-05
> > > > > > > > 153 KSP preconditioned resid norm 3.348992579120e-06 true
> resid norm 2.490655722697e-04 ||r(i)||/||b|| 3.381404735121e-05
> > > > > > > > 154 KSP preconditioned resid norm 3.309431137943e-06 true
> resid norm 2.490727563300e-04 ||r(i)||/||b|| 3.381502268535e-05
> > > > > > > > 155 KSP preconditioned resid norm 3.299031245428e-06 true
> resid norm 2.490688392843e-04 ||r(i)||/||b|| 3.381449089298e-05
> > > > > > > > 156 KSP preconditioned resid norm 3.297127463503e-06 true
> resid norm 2.490642207769e-04 ||r(i)||/||b|| 3.381386386763e-05
> > > > > > > > 157 KSP preconditioned resid norm 3.297370198641e-06 true
> resid norm 2.490666651723e-04 ||r(i)||/||b|| 3.381419572764e-05
> > > > > > > > 158 KSP preconditioned resid norm 3.290873165210e-06 true
> resid norm 2.490679189538e-04 ||r(i)||/||b|| 3.381436594557e-05
> > > > > > > > 159 KSP preconditioned resid norm 3.346705292419e-06 true
> resid norm 2.490617329776e-04 ||r(i)||/||b|| 3.381352611496e-05
> > > > > > > > 160 KSP preconditioned resid norm 3.429583550890e-06 true
> resid norm 2.490675116236e-04 ||r(i)||/||b|| 3.381431064494e-05
> > > > > > > > 161 KSP preconditioned resid norm 3.425238504679e-06 true
> resid norm 2.490648199058e-04 ||r(i)||/||b|| 3.381394520756e-05
> > > > > > > > 162 KSP preconditioned resid norm 3.423484857849e-06 true
> resid norm 2.490723208298e-04 ||r(i)||/||b|| 3.381496356025e-05
> > > > > > > > 163 KSP preconditioned resid norm 3.383655922943e-06 true
> resid norm 2.490659981249e-04 ||r(i)||/||b|| 3.381410516686e-05
> > > > > > > > 164 KSP preconditioned resid norm 3.477197358452e-06 true
> resid norm 2.490665979073e-04 ||r(i)||/||b|| 3.381418659549e-05
> > > > > > > > 165 KSP preconditioned resid norm 3.454672202601e-06 true
> resid norm 2.490651358644e-04 ||r(i)||/||b|| 3.381398810323e-05
> > > > > > > > 166 KSP preconditioned resid norm 3.399075522566e-06 true
> resid norm 2.490678159511e-04 ||r(i)||/||b|| 3.381435196154e-05
> > > > > > > > 167 KSP preconditioned resid norm 3.305455787400e-06 true
> resid norm 2.490651924523e-04 ||r(i)||/||b|| 3.381399578581e-05
> > > > > > > > 168 KSP preconditioned resid norm 3.368445533284e-06 true
> resid norm 2.490688061735e-04 ||r(i)||/||b|| 3.381448639774e-05
> > > > > > > > 169 KSP preconditioned resid norm 2.981519724814e-06 true
> resid norm 2.490676378334e-04 ||r(i)||/||b|| 3.381432777964e-05
> > > > > > > > 170 KSP preconditioned resid norm 3.034423065539e-06 true
> resid norm 2.490694458885e-04 ||r(i)||/||b|| 3.381457324777e-05
> > > > > > > > 171 KSP preconditioned resid norm 2.885972780503e-06 true
> resid norm 2.490688033729e-04 ||r(i)||/||b|| 3.381448601752e-05
> > > > > > > > 172 KSP preconditioned resid norm 2.892491075033e-06 true
> resid norm 2.490692993765e-04 ||r(i)||/||b|| 3.381455335678e-05
> > > > > > > > 173 KSP preconditioned resid norm 2.921316177611e-06 true
> resid norm 2.490697629787e-04 ||r(i)||/||b|| 3.381461629709e-05
> > > > > > > > 174 KSP preconditioned resid norm 2.999889222269e-06 true
> resid norm 2.490707272626e-04 ||r(i)||/||b|| 3.381474721178e-05
> > > > > > > > 175 KSP preconditioned resid norm 2.975590207575e-06 true
> resid norm 2.490685439925e-04 ||r(i)||/||b|| 3.381445080310e-05
> > > > > > > > 176 KSP preconditioned resid norm 2.983065843597e-06 true
> resid norm 2.490701883671e-04 ||r(i)||/||b|| 3.381467404937e-05
> > > > > > > > 177 KSP preconditioned resid norm 2.965959610245e-06 true
> resid norm 2.490711538630e-04 ||r(i)||/||b|| 3.381480512861e-05
> > > > > > > > 178 KSP preconditioned resid norm 3.005389788827e-06 true
> resid norm 2.490751808095e-04 ||r(i)||/||b|| 3.381535184150e-05
> > > > > > > > 179 KSP preconditioned resid norm 2.956581668772e-06 true
> resid norm 2.490653125636e-04 ||r(i)||/||b|| 3.381401209257e-05
> > > > > > > > 180 KSP preconditioned resid norm 2.937498883661e-06 true
> resid norm 2.490666056653e-04 ||r(i)||/||b|| 3.381418764874e-05
> > > > > > > > 181 KSP preconditioned resid norm 2.913227475431e-06 true
> resid norm 2.490682436979e-04 ||r(i)||/||b|| 3.381441003402e-05
> > > > > > > > 182 KSP preconditioned resid norm 3.048172862254e-06 true
> resid norm 2.490719669872e-04 ||r(i)||/||b|| 3.381491552130e-05
> > > > > > > > 183 KSP preconditioned resid norm 3.023868104933e-06 true
> resid norm 2.490648745555e-04 ||r(i)||/||b|| 3.381395262699e-05
> > > > > > > > 184 KSP preconditioned resid norm 2.985947506400e-06 true
> resid norm 2.490638818852e-04 ||r(i)||/||b|| 3.381381785846e-05
> > > > > > > > 185 KSP preconditioned resid norm 2.840032055776e-06 true
> resid norm 2.490701112392e-04 ||r(i)||/||b|| 3.381466357820e-05
> > > > > > > > 186 KSP preconditioned resid norm 2.229279683815e-06 true
> resid norm 2.490609220680e-04 ||r(i)||/||b|| 3.381341602292e-05
> > > > > > > > 187 KSP preconditioned resid norm 2.441513276379e-06 true
> resid norm 2.490674056899e-04 ||r(i)||/||b|| 3.381429626300e-05
> > > > > > > > 188 KSP preconditioned resid norm 2.467046864016e-06 true
> resid norm 2.490691622632e-04 ||r(i)||/||b|| 3.381453474178e-05
> > > > > > > > 189 KSP preconditioned resid norm 2.482124586361e-06 true
> resid norm 2.490664992339e-04 ||r(i)||/||b|| 3.381417319923e-05
> > > > > > > > 190 KSP preconditioned resid norm 2.470564926502e-06 true
> resid norm 2.490617019713e-04 ||r(i)||/||b|| 3.381352190543e-05
> > > > > > > > 191 KSP preconditioned resid norm 2.457947086578e-06 true
> resid norm 2.490628644250e-04 ||r(i)||/||b|| 3.381367972437e-05
> > > > > > > > 192 KSP preconditioned resid norm 2.469444741724e-06 true
> resid norm 2.490639416335e-04 ||r(i)||/||b|| 3.381382597011e-05
> > > > > > > > 193 KSP preconditioned resid norm 2.469951525219e-06 true
> resid norm 2.490599769764e-04 ||r(i)||/||b|| 3.381328771385e-05
> > > > > > > > 194 KSP preconditioned resid norm 2.467486786643e-06 true
> resid norm 2.490630178622e-04 ||r(i)||/||b|| 3.381370055556e-05
> > > > > > > > 195 KSP preconditioned resid norm 2.409684391404e-06 true
> resid norm 2.490640302606e-04 ||r(i)||/||b|| 3.381383800245e-05
> > > > > > > > 196 KSP preconditioned resid norm 2.456046691135e-06 true
> resid norm 2.490637730235e-04 ||r(i)||/||b|| 3.381380307900e-05
> > > > > > > > 197 KSP preconditioned resid norm 2.300015653805e-06 true
> resid norm 2.490615406913e-04 ||r(i)||/||b|| 3.381350000947e-05
> > > > > > > > 198 KSP preconditioned resid norm 2.238328275301e-06 true
> resid norm 2.490647641246e-04 ||r(i)||/||b|| 3.381393763449e-05
> > > > > > > > 199 KSP preconditioned resid norm 2.317293820319e-06 true
> resid norm 2.490641611282e-04 ||r(i)||/||b|| 3.381385576951e-05
> > > > > > > > 200 KSP preconditioned resid norm 2.359590971314e-06 true
> resid norm 2.490685242974e-04 ||r(i)||/||b|| 3.381444812922e-05
> > > > > > > > 201 KSP preconditioned resid norm 2.311199691596e-06 true
> resid norm 2.490656791753e-04 ||r(i)||/||b|| 3.381406186510e-05
> > > > > > > > 202 KSP preconditioned resid norm 2.328772904196e-06 true
> resid norm 2.490651045523e-04 ||r(i)||/||b|| 3.381398385220e-05
> > > > > > > > 203 KSP preconditioned resid norm 2.332731604717e-06 true
> resid norm 2.490649960574e-04 ||r(i)||/||b|| 3.381396912253e-05
> > > > > > > > 204 KSP preconditioned resid norm 2.357629383490e-06 true
> resid norm 2.490686317727e-04 ||r(i)||/||b|| 3.381446272046e-05
> > > > > > > > 205 KSP preconditioned resid norm 2.374856180299e-06 true
> resid norm 2.490645897176e-04 ||r(i)||/||b|| 3.381391395637e-05
> > > > > > > > 206 KSP preconditioned resid norm 2.340395514404e-06 true
> resid norm 2.490618341127e-04 ||r(i)||/||b|| 3.381353984542e-05
> > > > > > > > 207 KSP preconditioned resid norm 2.314963680954e-06 true
> resid norm 2.490676153984e-04 ||r(i)||/||b|| 3.381432473379e-05
> > > > > > > > 208 KSP preconditioned resid norm 2.448070953106e-06 true
> resid norm 2.490644606776e-04 ||r(i)||/||b|| 3.381389643743e-05
> > > > > > > > 209 KSP preconditioned resid norm 2.428805110632e-06 true
> resid norm 2.490635817597e-04 ||r(i)||/||b|| 3.381377711234e-05
> > > > > > > > 210 KSP preconditioned resid norm 2.537929937808e-06 true
> resid norm 2.490680589404e-04 ||r(i)||/||b|| 3.381438495066e-05
> > > > > > > > 211 KSP preconditioned resid norm 2.515909029682e-06 true
> resid norm 2.490687803038e-04 ||r(i)||/||b|| 3.381448288557e-05
> > > > > > > > 212 KSP preconditioned resid norm 2.497907513266e-06 true
> resid norm 2.490618016885e-04 ||r(i)||/||b|| 3.381353544340e-05
> > > > > > > > 213 KSP preconditioned resid norm 1.783501869502e-06 true
> resid norm 2.490632647470e-04 ||r(i)||/||b|| 3.381373407354e-05
> > > > > > > > 214 KSP preconditioned resid norm 1.767420653144e-06 true
> resid norm 2.490685569328e-04 ||r(i)||/||b|| 3.381445255992e-05
> > > > > > > > 215 KSP preconditioned resid norm 1.854926068272e-06 true
> resid norm 2.490609365464e-04 ||r(i)||/||b|| 3.381341798856e-05
> > > > > > > > 216 KSP preconditioned resid norm 1.818308539774e-06 true
> resid norm 2.490639142283e-04 ||r(i)||/||b|| 3.381382224948e-05
> > > > > > > > 217 KSP preconditioned resid norm 1.809431578070e-06 true
> resid norm 2.490605125049e-04 ||r(i)||/||b|| 3.381336041915e-05
> > > > > > > > 218 KSP preconditioned resid norm 1.789862735999e-06 true
> resid norm 2.490564024901e-04 ||r(i)||/||b|| 3.381280242859e-05
> > > > > > > > 219 KSP preconditioned resid norm 1.769239890163e-06 true
> resid norm 2.490647825316e-04 ||r(i)||/||b|| 3.381394013349e-05
> > > > > > > > 220 KSP preconditioned resid norm 1.780760773109e-06 true
> resid norm 2.490622606663e-04 ||r(i)||/||b|| 3.381359775589e-05
> > > > > > > > 221 KSP preconditioned resid norm 5.009024913368e-07 true
> resid norm 2.490659101637e-04 ||r(i)||/||b|| 3.381409322492e-05
> > > > > > > > 222 KSP preconditioned resid norm 4.974450322799e-07 true
> resid norm 2.490714287402e-04 ||r(i)||/||b|| 3.381484244693e-05
> > > > > > > > 223 KSP preconditioned resid norm 4.938819481519e-07 true
> resid norm 2.490665661715e-04 ||r(i)||/||b|| 3.381418228693e-05
> > > > > > > > 224 KSP preconditioned resid norm 4.973231831266e-07 true
> resid norm 2.490725000995e-04 ||r(i)||/||b|| 3.381498789855e-05
> > > > > > > > 225 KSP preconditioned resid norm 5.086864036771e-07 true
> resid norm 2.490664132954e-04 ||r(i)||/||b|| 3.381416153192e-05
> > > > > > > > 226 KSP preconditioned resid norm 5.046954570561e-07 true
> resid norm 2.490698772594e-04 ||r(i)||/||b|| 3.381463181226e-05
> > > > > > > > 227 KSP preconditioned resid norm 5.086852920874e-07 true
> resid norm 2.490703544723e-04 ||r(i)||/||b|| 3.381469660041e-05
> > > > > > > > 228 KSP preconditioned resid norm 5.182381756169e-07 true
> resid norm 2.490665200032e-04 ||r(i)||/||b|| 3.381417601896e-05
> > > > > > > > 229 KSP preconditioned resid norm 5.261455182896e-07 true
> resid norm 2.490697169472e-04 ||r(i)||/||b|| 3.381461004770e-05
> > > > > > > > 230 KSP preconditioned resid norm 5.265262522400e-07 true
> resid norm 2.490726890541e-04 ||r(i)||/||b|| 3.381501355172e-05
> > > > > > > > 231 KSP preconditioned resid norm 5.220652263946e-07 true
> resid norm 2.490689325236e-04 ||r(i)||/||b|| 3.381450355149e-05
> > > > > > > > 232 KSP preconditioned resid norm 5.256466259888e-07 true
> resid norm 2.490694033989e-04 ||r(i)||/||b|| 3.381456747923e-05
> > > > > > > > 233 KSP preconditioned resid norm 5.443022648374e-07 true
> resid norm 2.490650183144e-04 ||r(i)||/||b|| 3.381397214423e-05
> > > > > > > > 234 KSP preconditioned resid norm 5.562619006436e-07 true
> resid norm 2.490764576883e-04 ||r(i)||/||b|| 3.381552519520e-05
> > > > > > > > 235 KSP preconditioned resid norm 5.998148629545e-07 true
> resid norm 2.490714032716e-04 ||r(i)||/||b|| 3.381483898922e-05
> > > > > > > > 236 KSP preconditioned resid norm 6.498977322955e-07 true
> resid norm 2.490650270144e-04 ||r(i)||/||b|| 3.381397332537e-05
> > > > > > > > 237 KSP preconditioned resid norm 6.503686003429e-07 true
> resid norm 2.490706976108e-04 ||r(i)||/||b|| 3.381474318615e-05
> > > > > > > > 238 KSP preconditioned resid norm 6.566719023119e-07 true
> resid norm 2.490664107559e-04 ||r(i)||/||b|| 3.381416118714e-05
> > > > > > > > 239 KSP preconditioned resid norm 6.549737473208e-07 true
> resid norm 2.490721547909e-04 ||r(i)||/||b|| 3.381494101821e-05
> > > > > > > > 240 KSP preconditioned resid norm 6.616898981418e-07 true
> resid norm 2.490679659838e-04 ||r(i)||/||b|| 3.381437233053e-05
> > > > > > > > 241 KSP preconditioned resid norm 6.829917691021e-07 true
> resid norm 2.490728328614e-04 ||r(i)||/||b|| 3.381503307553e-05
> > > > > > > > 242 KSP preconditioned resid norm 7.030239869389e-07 true
> resid norm 2.490706345115e-04 ||r(i)||/||b|| 3.381473461955e-05
> > > > > > > > 243 KSP preconditioned resid norm 7.018435683340e-07 true
> resid norm 2.490650978460e-04 ||r(i)||/||b|| 3.381398294172e-05
> > > > > > > > 244 KSP preconditioned resid norm 7.058047080376e-07 true
> resid norm 2.490685975642e-04 ||r(i)||/||b|| 3.381445807618e-05
> > > > > > > > 245 KSP preconditioned resid norm 6.896300385099e-07 true
> resid norm 2.490708566380e-04 ||r(i)||/||b|| 3.381476477625e-05
> > > > > > > > 246 KSP preconditioned resid norm 7.093960074437e-07 true
> resid norm 2.490667427871e-04 ||r(i)||/||b|| 3.381420626490e-05
> > > > > > > > 247 KSP preconditioned resid norm 7.817121711853e-07 true
> resid norm 2.490692299030e-04 ||r(i)||/||b|| 3.381454392480e-05
> > > > > > > > 248 KSP preconditioned resid norm 7.976109778309e-07 true
> resid norm 2.490686360729e-04 ||r(i)||/||b|| 3.381446330426e-05
> > > > > > > > 249 KSP preconditioned resid norm 7.855322750445e-07 true
> resid norm 2.490720861966e-04 ||r(i)||/||b|| 3.381493170560e-05
> > > > > > > > 250 KSP preconditioned resid norm 7.778531114042e-07 true
> resid norm 2.490673235034e-04 ||r(i)||/||b|| 3.381428510506e-05
> > > > > > > > 251 KSP preconditioned resid norm 7.848682182070e-07 true
> resid norm 2.490686360729e-04 ||r(i)||/||b|| 3.381446330426e-05
> > > > > > > > 252 KSP preconditioned resid norm 7.967291867330e-07 true
> resid norm 2.490724820229e-04 ||r(i)||/||b|| 3.381498544442e-05
> > > > > > > > 253 KSP preconditioned resid norm 7.865012959525e-07 true
> resid norm 2.490666662028e-04 ||r(i)||/||b|| 3.381419586754e-05
> > > > > > > > 254 KSP preconditioned resid norm 7.656025385804e-07 true
> resid norm 2.490686283214e-04 ||r(i)||/||b|| 3.381446225190e-05
> > > > > > > > 255 KSP preconditioned resid norm 7.757018653468e-07 true
> resid norm 2.490655983763e-04 ||r(i)||/||b|| 3.381405089553e-05
> > > > > > > > 256 KSP preconditioned resid norm 6.686490372981e-07 true
> resid norm 2.490715698964e-04 ||r(i)||/||b|| 3.381486161081e-05
> > > > > > > > 257 KSP preconditioned resid norm 6.596005109428e-07 true
> resid norm 2.490666403003e-04 ||r(i)||/||b|| 3.381419235092e-05
> > > > > > > > 258 KSP preconditioned resid norm 6.681742296333e-07 true
> resid norm 2.490683725835e-04 ||r(i)||/||b|| 3.381442753198e-05
> > > > > > > > 259 KSP preconditioned resid norm 1.089245482033e-06 true
> resid norm 2.490688086568e-04 ||r(i)||/||b|| 3.381448673488e-05
> > > > > > > > 260 KSP preconditioned resid norm 1.099844873189e-06 true
> resid norm 2.490690703265e-04 ||r(i)||/||b|| 3.381452226011e-05
> > > > > > > > 261 KSP preconditioned resid norm 1.112925540869e-06 true
> resid norm 2.490664481058e-04 ||r(i)||/||b|| 3.381416625790e-05
> > > > > > > > 262 KSP preconditioned resid norm 1.113056910480e-06 true
> resid norm 2.490658753273e-04 ||r(i)||/||b|| 3.381408849541e-05
> > > > > > > > 263 KSP preconditioned resid norm 1.104801535149e-06 true
> resid norm 2.490736510776e-04 ||r(i)||/||b|| 3.381514415953e-05
> > > > > > > > 264 KSP preconditioned resid norm 1.158709147873e-06 true
> resid norm 2.490607531152e-04 ||r(i)||/||b|| 3.381339308528e-05
> > > > > > > > 265 KSP preconditioned resid norm 1.178985740182e-06 true
> resid norm 2.490727895619e-04 ||r(i)||/||b|| 3.381502719703e-05
> > > > > > > > 266 KSP preconditioned resid norm 1.165130533478e-06 true
> resid norm 2.490639076693e-04 ||r(i)||/||b|| 3.381382135901e-05
> > > > > > > > 267 KSP preconditioned resid norm 1.181364114499e-06 true
> resid norm 2.490667871436e-04 ||r(i)||/||b|| 3.381421228690e-05
> > > > > > > > 268 KSP preconditioned resid norm 1.170295348543e-06 true
> resid norm 2.490662613306e-04 ||r(i)||/||b|| 3.381414090063e-05
> > > > > > > > 269 KSP preconditioned resid norm 1.213243016230e-06 true
> resid norm 2.490666173719e-04 ||r(i)||/||b|| 3.381418923808e-05
> > > > > > > > 270 KSP preconditioned resid norm 1.239691953997e-06 true
> resid norm 2.490678323197e-04 ||r(i)||/||b|| 3.381435418381e-05
> > > > > > > > 271 KSP preconditioned resid norm 1.219891740100e-06 true
> resid norm 2.490625009256e-04 ||r(i)||/||b|| 3.381363037437e-05
> > > > > > > > 272 KSP preconditioned resid norm 1.231321334346e-06 true
> resid norm 2.490659733696e-04 ||r(i)||/||b|| 3.381410180599e-05
> > > > > > > > 273 KSP preconditioned resid norm 1.208183234158e-06 true
> resid norm 2.490685987255e-04 ||r(i)||/||b|| 3.381445823385e-05
> > > > > > > > 274 KSP preconditioned resid norm 1.211768545589e-06 true
> resid norm 2.490671548953e-04 ||r(i)||/||b|| 3.381426221421e-05
> > > > > > > > 275 KSP preconditioned resid norm 1.209433459842e-06 true
> resid norm 2.490669016096e-04 ||r(i)||/||b|| 3.381422782722e-05
> > > > > > > > 276 KSP preconditioned resid norm 1.223729184405e-06 true
> resid norm 2.490658128014e-04 ||r(i)||/||b|| 3.381408000666e-05
> > > > > > > > 277 KSP preconditioned resid norm 1.243915201868e-06 true
> resid norm 2.490693375756e-04 ||r(i)||/||b|| 3.381455854282e-05
> > > > > > > > 278 KSP preconditioned resid norm 1.231994655529e-06 true
> resid norm 2.490682988311e-04 ||r(i)||/||b|| 3.381441751910e-05
> > > > > > > > 279 KSP preconditioned resid norm 1.227930683777e-06 true
> resid norm 2.490667825866e-04 ||r(i)||/||b|| 3.381421166823e-05
> > > > > > > > 280 KSP preconditioned resid norm 1.193458846469e-06 true
> resid norm 2.490687366117e-04 ||r(i)||/||b|| 3.381447695378e-05
> > > > > > > > 281 KSP preconditioned resid norm 1.217089059805e-06 true
> resid norm 2.490674797371e-04 ||r(i)||/||b|| 3.381430631591e-05
> > > > > > > > 282 KSP preconditioned resid norm 1.249318287709e-06 true
> resid norm 2.490662866951e-04 ||r(i)||/||b|| 3.381414434420e-05
> > > > > > > > 283 KSP preconditioned resid norm 1.183320029547e-06 true
> resid norm 2.490645783630e-04 ||r(i)||/||b|| 3.381391241482e-05
> > > > > > > > 284 KSP preconditioned resid norm 1.174730603102e-06 true
> resid norm 2.490686881647e-04 ||r(i)||/||b|| 3.381447037643e-05
> > > > > > > > 285 KSP preconditioned resid norm 1.175838261923e-06 true
> resid norm 2.490665969300e-04 ||r(i)||/||b|| 3.381418646281e-05
> > > > > > > > 286 KSP preconditioned resid norm 1.188946188368e-06 true
> resid norm 2.490661974622e-04 ||r(i)||/||b|| 3.381413222961e-05
> > > > > > > > 287 KSP preconditioned resid norm 1.177848565707e-06 true
> resid norm 2.490660236206e-04 ||r(i)||/||b|| 3.381410862824e-05
> > > > > > > > 288 KSP preconditioned resid norm 1.200075508281e-06 true
> resid norm 2.490645353536e-04 ||r(i)||/||b|| 3.381390657571e-05
> > > > > > > > 289 KSP preconditioned resid norm 1.184589570618e-06 true
> resid norm 2.490664920355e-04 ||r(i)||/||b|| 3.381417222195e-05
> > > > > > > > 290 KSP preconditioned resid norm 1.221114703873e-06 true
> resid norm 2.490670597538e-04 ||r(i)||/||b|| 3.381424929746e-05
> > > > > > > > 291 KSP preconditioned resid norm 1.249479658256e-06 true
> resid norm 2.490641582876e-04 ||r(i)||/||b|| 3.381385538385e-05
> > > > > > > > 292 KSP preconditioned resid norm 1.245768496850e-06 true
> resid norm 2.490704480588e-04 ||r(i)||/||b|| 3.381470930606e-05
> > > > > > > > 293 KSP preconditioned resid norm 1.243742607953e-06 true
> resid norm 2.490649690604e-04 ||r(i)||/||b|| 3.381396545733e-05
> > > > > > > > 294 KSP preconditioned resid norm 1.342758483339e-06 true
> resid norm 2.490676207432e-04 ||r(i)||/||b|| 3.381432545942e-05
> > > > > > > > 295 KSP preconditioned resid norm 1.353816099600e-06 true
> resid norm 2.490695263153e-04 ||r(i)||/||b|| 3.381458416681e-05
> > > > > > > > 296 KSP preconditioned resid norm 1.343886763293e-06 true
> resid norm 2.490673674307e-04 ||r(i)||/||b|| 3.381429106879e-05
> > > > > > > > 297 KSP preconditioned resid norm 1.355511022815e-06 true
> resid norm 2.490686565995e-04 ||r(i)||/||b|| 3.381446609103e-05
> > > > > > > > 298 KSP preconditioned resid norm 1.347247627243e-06 true
> resid norm 2.490696287707e-04 ||r(i)||/||b|| 3.381459807652e-05
> > > > > > > > 299 KSP preconditioned resid norm 1.414742595618e-06 true
> resid norm 2.490749815091e-04 ||r(i)||/||b|| 3.381532478374e-05
> > > > > > > > 300 KSP preconditioned resid norm 1.418560683189e-06 true
> resid norm 2.490721501153e-04 ||r(i)||/||b|| 3.381494038343e-05
> > > > > > > > 301 KSP preconditioned resid norm 1.416276404923e-06 true
> resid norm 2.490689576447e-04 ||r(i)||/||b|| 3.381450696203e-05
> > > > > > > > 302 KSP preconditioned resid norm 1.431448272112e-06 true
> resid norm 2.490688812701e-04 ||r(i)||/||b|| 3.381449659312e-05
> > > > > > > > 303 KSP preconditioned resid norm 1.446154958969e-06 true
> resid norm 2.490727536322e-04 ||r(i)||/||b|| 3.381502231909e-05
> > > > > > > > 304 KSP preconditioned resid norm 1.468860617921e-06 true
> resid norm 2.490692363788e-04 ||r(i)||/||b|| 3.381454480397e-05
> > > > > > > > 305 KSP preconditioned resid norm 1.627595214971e-06 true
> resid norm 2.490687019603e-04 ||r(i)||/||b|| 3.381447224938e-05
> > > > > > > > 306 KSP preconditioned resid norm 1.614384672893e-06 true
> resid norm 2.490687019603e-04 ||r(i)||/||b|| 3.381447224938e-05
> > > > > > > > 307 KSP preconditioned resid norm 1.605568020532e-06 true
> resid norm 2.490699757693e-04 ||r(i)||/||b|| 3.381464518632e-05
> > > > > > > > 308 KSP preconditioned resid norm 1.617069685075e-06 true
> resid norm 2.490649282923e-04 ||r(i)||/||b|| 3.381395992249e-05
> > > > > > > > 309 KSP preconditioned resid norm 1.654297792738e-06 true
> resid norm 2.490644766626e-04 ||r(i)||/||b|| 3.381389860760e-05
> > > > > > > > 310 KSP preconditioned resid norm 1.587528143215e-06 true
> resid norm 2.490696752096e-04 ||r(i)||/||b|| 3.381460438124e-05
> > > > > > > > 311 KSP preconditioned resid norm 1.662782022388e-06 true
> resid norm 2.490699317737e-04 ||r(i)||/||b|| 3.381463921332e-05
> > > > > > > > 312 KSP preconditioned resid norm 1.618211471748e-06 true
> resid norm 2.490735831308e-04 ||r(i)||/||b|| 3.381513493483e-05
> > > > > > > > 313 KSP preconditioned resid norm 1.609074961921e-06 true
> resid norm 2.490679566436e-04 ||r(i)||/||b|| 3.381437106247e-05
> > > > > > > > 314 KSP preconditioned resid norm 1.548068942878e-06 true
> resid norm 2.490660071226e-04 ||r(i)||/||b|| 3.381410638842e-05
> > > > > > > > 315 KSP preconditioned resid norm 1.526718322150e-06 true
> resid norm 2.490619832967e-04 ||r(i)||/||b|| 3.381356009919e-05
> > > > > > > > 316 KSP preconditioned resid norm 1.553150959105e-06 true
> resid norm 2.490660071226e-04 ||r(i)||/||b|| 3.381410638842e-05
> > > > > > > > 317 KSP preconditioned resid norm 1.615015320906e-06 true
> resid norm 2.490672348079e-04 ||r(i)||/||b|| 3.381427306343e-05
> > > > > > > > 318 KSP preconditioned resid norm 1.602904469797e-06 true
> resid norm 2.490696731006e-04 ||r(i)||/||b|| 3.381460409491e-05
> > > > > > > > 319 KSP preconditioned resid norm 1.538140323073e-06 true
> resid norm 2.490722982494e-04 ||r(i)||/||b|| 3.381496049466e-05
> > > > > > > > 320 KSP preconditioned resid norm 1.534779679430e-06 true
> resid norm 2.490778499789e-04 ||r(i)||/||b|| 3.381571421763e-05
> > > > > > > > 321 KSP preconditioned resid norm 1.547155843355e-06 true
> resid norm 2.490767612985e-04 ||r(i)||/||b|| 3.381556641442e-05
> > > > > > > > 322 KSP preconditioned resid norm 1.422137008870e-06 true
> resid norm 2.490737676309e-04 ||r(i)||/||b|| 3.381515998323e-05
> > > > > > > > 323 KSP preconditioned resid norm 1.403072558954e-06 true
> resid norm 2.490741361870e-04 ||r(i)||/||b|| 3.381521001975e-05
> > > > > > > > 324 KSP preconditioned resid norm 1.373070436118e-06 true
> resid norm 2.490742214990e-04 ||r(i)||/||b|| 3.381522160202e-05
> > > > > > > > 325 KSP preconditioned resid norm 1.359547585233e-06 true
> resid norm 2.490792987570e-04 ||r(i)||/||b|| 3.381591090902e-05
> > > > > > > > 326 KSP preconditioned resid norm 1.370351913612e-06 true
> resid norm 2.490727161158e-04 ||r(i)||/||b|| 3.381501722573e-05
> > > > > > > > 327 KSP preconditioned resid norm 1.365238666187e-06 true
> resid norm 2.490716949642e-04 ||r(i)||/||b|| 3.381487859046e-05
> > > > > > > > 328 KSP preconditioned resid norm 1.369073373042e-06 true
> resid norm 2.490807360288e-04 ||r(i)||/||b|| 3.381610603826e-05
> > > > > > > > 329 KSP preconditioned resid norm 1.426698981572e-06 true
> resid norm 2.490791479521e-04 ||r(i)||/||b|| 3.381589043520e-05
> > > > > > > > 330 KSP preconditioned resid norm 1.445542403570e-06 true
> resid norm 2.490775981409e-04 ||r(i)||/||b|| 3.381568002720e-05
> > > > > > > > 331 KSP preconditioned resid norm 1.464506963984e-06 true
> resid norm 2.490740562430e-04 ||r(i)||/||b|| 3.381519916626e-05
> > > > > > > > 332 KSP preconditioned resid norm 1.461462964401e-06 true
> resid norm 2.490768016856e-04 ||r(i)||/||b|| 3.381557189753e-05
> > > > > > > > 333 KSP preconditioned resid norm 1.476680847971e-06 true
> resid norm 2.490744321516e-04 ||r(i)||/||b|| 3.381525020097e-05
> > > > > > > > 334 KSP preconditioned resid norm 1.459640372198e-06 true
> resid norm 2.490788817993e-04 ||r(i)||/||b|| 3.381585430132e-05
> > > > > > > > 335 KSP preconditioned resid norm 1.790770882365e-06 true
> resid norm 2.490771711471e-04 ||r(i)||/||b|| 3.381562205697e-05
> > > > > > > > 336 KSP preconditioned resid norm 1.803770155018e-06 true
> resid norm 2.490768953858e-04 ||r(i)||/||b|| 3.381558461860e-05
> > > > > > > > 337 KSP preconditioned resid norm 1.787821255995e-06 true
> resid norm 2.490767985676e-04 ||r(i)||/||b|| 3.381557147421e-05
> > > > > > > > 338 KSP preconditioned resid norm 1.749912220831e-06 true
> resid norm 2.490760198704e-04 ||r(i)||/||b|| 3.381546575545e-05
> > > > > > > > 339 KSP preconditioned resid norm 1.802915839010e-06 true
> resid norm 2.490815556273e-04 ||r(i)||/||b|| 3.381621730993e-05
> > > > > > > > 340 KSP preconditioned resid norm 1.800777670709e-06 true
> resid norm 2.490823909286e-04 ||r(i)||/||b|| 3.381633071347e-05
> > > > > > > > 341 KSP preconditioned resid norm 1.962516327690e-06 true
> resid norm 2.490773477410e-04 ||r(i)||/||b|| 3.381564603199e-05
> > > > > > > > 342 KSP preconditioned resid norm 1.981726465132e-06 true
> resid norm 2.490769116884e-04 ||r(i)||/||b|| 3.381558683191e-05
> > > > > > > > 343 KSP preconditioned resid norm 1.963419167052e-06 true
> resid norm 2.490764009914e-04 ||r(i)||/||b|| 3.381551749783e-05
> > > > > > > > 344 KSP preconditioned resid norm 1.992082169278e-06 true
> resid norm 2.490806829883e-04 ||r(i)||/||b|| 3.381609883728e-05
> > > > > > > > 345 KSP preconditioned resid norm 1.981005134253e-06 true
> resid norm 2.490748677339e-04 ||r(i)||/||b|| 3.381530933721e-05
> > > > > > > > 346 KSP preconditioned resid norm 1.959802663114e-06 true
> resid norm 2.490773752317e-04 ||r(i)||/||b|| 3.381564976423e-05
> > > > > > > >
> > > > > > > > On Sat, Oct 21, 2017 at 5:25 PM, Matthew Knepley <
> knepley at gmail.com> wrote:
> > > > > > > > On Sat, Oct 21, 2017 at 5:21 PM, Hao Zhang <
> hbcbh1999 at gmail.com> wrote:
> > > > > > > > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
> > > > > > > > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
> > > > > > > >
> > > > > > > > ierr = VecAssemblyBegin(x);
> > > > > > > > ierr = VecAssemblyEnd(x);
> > > > > > > > This is probably unnecessary
> > > > > > > >
> > > > > > > > ierr = VecAssemblyBegin(b);
> > > > > > > > ierr = VecAssemblyEnd(b);
> > > > > > > > This is probably unnecessary
> > > > > > > >
> > > > > > > >
> > > > > > > > ierr = MatNullSpaceCreate(PETSC_COMM_
> WORLD,PETSC_TRUE,0,PETSC_NULL,&nullsp);
> > > > > > > > ierr = MatSetNullSpace(A,nullsp); // Petsc-3.8
> > > > > > > > Is your rhs consistent with this nullspace?
> > > > > > > >
> > > > > > > > // KSPSetOperators(ksp,A,A,
> DIFFERENT_NONZERO_PATTERN);
> > > > > > > > KSPSetOperators(ksp,A,A);
> > > > > > > >
> > > > > > > > KSPSetType(ksp,KSPBCGS);
> > > > > > > >
> > > > > > > > KSPSetComputeSingularValues(ksp, PETSC_TRUE);
> > > > > > > > #if defined(__HYPRE__)
> > > > > > > > KSPGetPC(ksp, &pc);
> > > > > > > > PCSetType(pc, PCHYPRE);
> > > > > > > > PCHYPRESetType(pc,"boomeramg");
> > > > > > > > This is terribly unnecessary. You just use
> > > > > > > >
> > > > > > > > -pc_type hypre -pc_hypre_type boomeramg
> > > > > > > >
> > > > > > > > or
> > > > > > > >
> > > > > > > > -pc_type gamg
> > > > > > > >
> > > > > > > > #else
> > > > > > > > KSPSetType(ksp,KSPBCGSL);
> > > > > > > > KSPBCGSLSetEll(ksp,2);
> > > > > > > > #endif /* defined(__HYPRE__) */
> > > > > > > >
> > > > > > > > KSPSetFromOptions(ksp);
> > > > > > > > KSPSetUp(ksp);
> > > > > > > >
> > > > > > > > ierr = KSPSolve(ksp,b,x);
> > > > > > > >
> > > > > > > >
> > > > > > > > command line
> > > > > > > >
> > > > > > > > You did not provide any of what I asked for the in the
> eprevious mail.
> > > > > > > >
> > > > > > > > Matt
> > > > > > > >
> > > > > > > > On Sat, Oct 21, 2017 at 5:16 PM, Matthew Knepley <
> knepley at gmail.com> wrote:
> > > > > > > > On Sat, Oct 21, 2017 at 5:04 PM, Hao Zhang <
> hbcbh1999 at gmail.com> wrote:
> > > > > > > > hi,
> > > > > > > >
> > > > > > > > I implemented HYPRE preconditioner for my study due to the
> fact that without preconditioner, PETSc solver will take thousands of
> iterations to converge for fine grid simulation.
> > > > > > > >
> > > > > > > > with HYPRE, depending on the parallel partition, it will
> take HYPRE forever to do anything. observation of output file is that the
> simulation is hanging with no output.
> > > > > > > >
> > > > > > > > Any idea what happened? will post snippet of code.
> > > > > > > >
> > > > > > > > 1) For any question about convergence, we need to see the
> output of
> > > > > > > >
> > > > > > > > -ksp_view_pre -ksp_view -ksp_monitor_true_residual
> -ksp_converged_reason
> > > > > > > >
> > > > > > > > 2) Hypre has many preconditioners, which one are you talking
> about
> > > > > > > >
> > > > > > > > 3) PETSc has some preconditioners in common with Hypre, like
> AMG
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Matt
> > > > > > > >
> > > > > > > > --
> > > > > > > > Hao Zhang
> > > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > > Stony Brook University,
> > > > > > > > Stony Brook, New York, 11790
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > What most experimenters take for granted before they begin
> their experiments is infinitely more interesting than any results to which
> their experiments lead.
> > > > > > > > -- Norbert Wiener
> > > > > > > >
> > > > > > > > https://www.cse.buffalo.edu/~knepley/
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Hao Zhang
> > > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > > Stony Brook University,
> > > > > > > > Stony Brook, New York, 11790
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > What most experimenters take for granted before they begin
> their experiments is infinitely more interesting than any results to which
> their experiments lead.
> > > > > > > > -- Norbert Wiener
> > > > > > > >
> > > > > > > > https://www.cse.buffalo.edu/~knepley/
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Hao Zhang
> > > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > > Stony Brook University,
> > > > > > > > Stony Brook, New York, 11790
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Hao Zhang
> > > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > > Stony Brook University,
> > > > > > > > Stony Brook, New York, 11790
> > > > > > >
> > > > > > > --
> > > > > > > Hao Zhang
> > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > Stony Brook University,
> > > > > > > Stony Brook, New York, 11790
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Hao Zhang
> > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > Stony Brook University,
> > > > > > Stony Brook, New York, 11790
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Hao Zhang
> > > > > Dept. of Applid Mathematics and Statistics,
> > > > > Stony Brook University,
> > > > > Stony Brook, New York, 11790
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Hao Zhang
> > > > Dept. of Applid Mathematics and Statistics,
> > > > Stony Brook University,
> > > > Stony Brook, New York, 11790
> > >
> > >
> > >
> > >
> > > --
> > > Hao Zhang
> > > Dept. of Applid Mathematics and Statistics,
> > > Stony Brook University,
> > > Stony Brook, New York, 11790
> >
> >
> >
> >
> > --
> > Hao Zhang
> > Dept. of Applid Mathematics and Statistics,
> > Stony Brook University,
> > Stony Brook, New York, 11790
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171023/64e26fe7/attachment-0001.html>
More information about the petsc-users
mailing list