[petsc-users] HYPRE hanging or slow? from observation
Barry Smith
bsmith at mcs.anl.gov
Mon Oct 23 10:36:08 CDT 2017
I'm confused. Is hypre + GMRES ever not working fine for you? Why not just always use hypre + gmres; no basic solver is going to be faster for large problems ever.
Barry
> On Oct 23, 2017, at 9:09 AM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
>
> Yes.
>
> On Mon, Oct 23, 2017 at 10:07 AM, Mark Adams <mfadams at lbl.gov> wrote:
>
>
> On Mon, Oct 23, 2017 at 10:04 AM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> The big picture is I'm solving 3d incompressible Navier-Stokes Equations using staggered/MAC grid with Finite Difference Method. The particular function is poisson pressure solver or Laplacian Mark Adam mentioned. The simulation runs fine for medium size mesh grid. When I try harder to go very fine grid, not DNS level, I'm having some difficulties to have meaning physical results. All PETSc solvers converge but come with huge iterations. That's when/how I started using HYPRE.
>
> Are we just talking about the convergence of the pressure solve?
>
>
> On Mon, Oct 23, 2017 at 9:01 AM, Mark Adams <mfadams at lbl.gov> wrote:
> Just to be clear: 1) are you solving the Laplacian (div grad) and 2) what type of discretizations are you using? and 3) do you have stretched or terrible grids in some way?
>
> On Sun, Oct 22, 2017 at 3:57 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> One thing important to understand is that multigrid is an optimal or nearly optimal algorithm. This means, when it works, as you refine the mesh the number of iterations remains nearly constant, regardless of the problem size and number of processes. Simple preconditioners such as ILU, block Jacobi, one level additive Schwarz etc have iterations that increase with the problem size and likely also with the number of processes. Thus these algorithms become essentially impractical for very large problems while multigrid can remain practical (when it works).
>
> Good luck
>
> Barry
> > On Oct 22, 2017, at 2:35 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> >
> > Thanks for all the inputs. before simulating finer grid, HYPRE wasn't used and simulations were just fine. I will do a few tests and post more information later.
> >
> > On Sun, Oct 22, 2017 at 12:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Oct 21, 2017, at 11:16 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > >
> > > the reason is when I do finer grid simulation, matrix become more stiff.
> >
> > Are you saying that for a finer grid but everything else the same, the convergence of hypre (with the same GMRES) with the same options gets much worse? This normally will not happen, that is the fundamental beauty of multigrid methods (when they work well).
> >
> > Yes the matrix condition number increases but multigrid doesn't care about that, its number of iterations should remain pretty much the same.
> >
> > Something must be different (with this finer grid case), either the mesh becomes horrible, or the physics changes, or there are errors in the code that lead to the problem.
> >
> > What happens if you just refine the mesh a little? Then a little more? Then a little more? Does the convergence rate suddenly go bad at some point, or does it just get worse slowly?
> >
> > Barry
> >
> >
> >
> > > Much larger condition number. just to give you a perspective, it will take 6000 iterations to converge and the solver does converge. I want to reduce the number of iterations while keeping the convergence rate. that's main drive to do so much heavy lifting around. please advise. snippet will be provided upon request.
> > >
> > > Thanks again.
> > >
> > > On Sun, Oct 22, 2017 at 12:08 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >
> > > Oh, you change KSP but not hypre. I did not understand this.
> > >
> > > Why not just use GMRES all the time? Why mess with BCGS if it is not robust? Not worth the small optimization if it breaks everything.
> > >
> > > Barry
> > >
> > >
> > >
> > > > On Oct 21, 2017, at 11:05 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > >
> > > > this is the initial pressure solver output regarding use of PETSc. it failed to converge after 40000 iterations, then use GMRES.
> > > >
> > > > 39987 KSP preconditioned resid norm 3.853125986269e-08 true resid norm 1.147359212216e-05 ||r(i)||/||b|| 1.557696568706e-06
> > > > 39988 KSP preconditioned resid norm 3.853126044003e-08 true resid norm 1.147359257282e-05 ||r(i)||/||b|| 1.557696629889e-06
> > > > 39989 KSP preconditioned resid norm 3.853126052100e-08 true resid norm 1.147359233695e-05 ||r(i)||/||b|| 1.557696597866e-06
> > > > 39990 KSP preconditioned resid norm 3.853126027357e-08 true resid norm 1.147359219860e-05 ||r(i)||/||b|| 1.557696579083e-06
> > > > 39991 KSP preconditioned resid norm 3.853126058478e-08 true resid norm 1.147359234281e-05 ||r(i)||/||b|| 1.557696598662e-06
> > > > 39992 KSP preconditioned resid norm 3.853126064006e-08 true resid norm 1.147359261420e-05 ||r(i)||/||b|| 1.557696635506e-06
> > > > 39993 KSP preconditioned resid norm 3.853126050203e-08 true resid norm 1.147359235972e-05 ||r(i)||/||b|| 1.557696600957e-06
> > > > 39994 KSP preconditioned resid norm 3.853126050182e-08 true resid norm 1.147359253713e-05 ||r(i)||/||b|| 1.557696625043e-06
> > > > 39995 KSP preconditioned resid norm 3.853125976795e-08 true resid norm 1.147359226222e-05 ||r(i)||/||b|| 1.557696587720e-06
> > > > 39996 KSP preconditioned resid norm 3.853125805127e-08 true resid norm 1.147359262747e-05 ||r(i)||/||b|| 1.557696637308e-06
> > > > 39997 KSP preconditioned resid norm 3.853125811756e-08 true resid norm 1.147359216008e-05 ||r(i)||/||b|| 1.557696573853e-06
> > > > 39998 KSP preconditioned resid norm 3.853125827833e-08 true resid norm 1.147359238372e-05 ||r(i)||/||b|| 1.557696604216e-06
> > > > 39999 KSP preconditioned resid norm 3.853127937068e-08 true resid norm 1.147359264043e-05 ||r(i)||/||b|| 1.557696639067e-06
> > > > 40000 KSP preconditioned resid norm 3.853122732867e-08 true resid norm 1.147359257776e-05 ||r(i)||/||b|| 1.557696630559e-06
> > > > Linear solve did not converge due to DIVERGED_ITS iterations 40000
> > > > KSP Object: 24 MPI processes
> > > > type: bcgs
> > > > maximum iterations=40000, initial guess is zero
> > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > left preconditioning
> > > > using PRECONDITIONED norm type for convergence test
> > > > PC Object: 24 MPI processes
> > > > type: hypre
> > > > HYPRE BoomerAMG preconditioning
> > > > Cycle type V
> > > > Maximum number of levels 25
> > > > Maximum number of iterations PER hypre call 1
> > > > Convergence tolerance PER hypre call 0.
> > > > Threshold for strong coupling 0.25
> > > > Interpolation truncation factor 0.
> > > > Interpolation: max elements per row 0
> > > > Number of levels of aggressive coarsening 0
> > > > Number of paths for aggressive coarsening 1
> > > > Maximum row sums 0.9
> > > > Sweeps down 1
> > > > Sweeps up 1
> > > > Sweeps on coarse 1
> > > > Relax down symmetric-SOR/Jacobi
> > > > Relax up symmetric-SOR/Jacobi
> > > > Relax on coarse Gaussian-elimination
> > > > Relax weight (all) 1.
> > > > Outer relax weight (all) 1.
> > > > Using CF-relaxation
> > > > Not using more complex smoothers.
> > > > Measure type local
> > > > Coarsen type Falgout
> > > > Interpolation type classical
> > > > linear system matrix = precond matrix:
> > > > Mat Object: A 24 MPI processes
> > > > type: mpiaij
> > > > rows=497664, cols=497664
> > > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > > total number of mallocs used during MatSetValues calls =0
> > > > has attached null space
> > > > not using I-node (on process 0) routines
> > > >
> > > > The solution diverges for p0! The residual is 3.853123e-08. Solve again using GMRES!
> > > > KSP Object: 24 MPI processes
> > > > type: gmres
> > > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> > > > happy breakdown tolerance 1e-30
> > > > maximum iterations=40000, initial guess is zero
> > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > left preconditioning
> > > > using PRECONDITIONED norm type for convergence test
> > > > PC Object: 24 MPI processes
> > > > type: hypre
> > > > HYPRE BoomerAMG preconditioning
> > > > Cycle type V
> > > > Maximum number of levels 25
> > > > Maximum number of iterations PER hypre call 1
> > > > Convergence tolerance PER hypre call 0.
> > > > Threshold for strong coupling 0.25
> > > > Interpolation truncation factor 0.
> > > > Interpolation: max elements per row 0
> > > > Number of levels of aggressive coarsening 0
> > > > Number of paths for aggressive coarsening 1
> > > > Maximum row sums 0.9
> > > > Sweeps down 1
> > > > Sweeps up 1
> > > > Sweeps on coarse 1
> > > > Relax down symmetric-SOR/Jacobi
> > > > Relax up symmetric-SOR/Jacobi
> > > > Relax on coarse Gaussian-elimination
> > > > Relax weight (all) 1.
> > > > Outer relax weight (all) 1.
> > > > Using CF-relaxation
> > > > Not using more complex smoothers.
> > > > Measure type local
> > > > Coarsen type Falgout
> > > > Interpolation type classical
> > > > linear system matrix = precond matrix:
> > > > Mat Object: A 24 MPI processes
> > > > type: mpiaij
> > > > rows=497664, cols=497664
> > > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > > total number of mallocs used during MatSetValues calls =0
> > > > has attached null space
> > > > not using I-node (on process 0) routines
> > > > 0 KSP preconditioned resid norm 1.593802941804e+01 true resid norm 7.365742695119e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > > 1 KSP preconditioned resid norm 6.338666661133e-01 true resid norm 2.880722209358e+00 ||r(i)||/||b|| 3.910973174867e-01
> > > > 2 KSP preconditioned resid norm 3.913420828350e-02 true resid norm 9.544089760671e-01 ||r(i)||/||b|| 1.295740315093e-01
> > > > 3 KSP preconditioned resid norm 2.928070366435e-03 true resid norm 1.874294004628e-01 ||r(i)||/||b|| 2.544609664236e-02
> > > > 4 KSP preconditioned resid norm 2.165607525823e-04 true resid norm 3.121122463949e-02 ||r(i)||/||b|| 4.237349298146e-03
> > > > 5 KSP preconditioned resid norm 1.635476480407e-05 true resid norm 3.984315313831e-03 ||r(i)||/||b|| 5.409251284967e-04
> > > > 6 KSP preconditioned resid norm 1.283358350575e-06 true resid norm 4.566583802915e-04 ||r(i)||/||b|| 6.199760149022e-05
> > > > 7 KSP preconditioned resid norm 8.479469225747e-08 true resid norm 3.824581791112e-05 ||r(i)||/||b|| 5.192391248810e-06
> > > > 8 KSP preconditioned resid norm 5.358636504683e-09 true resid norm 2.829730442033e-06 ||r(i)||/||b|| 3.841744898187e-07
> > > > 9 KSP preconditioned resid norm 3.447874504193e-10 true resid norm 1.756617036538e-07 ||r(i)||/||b|| 2.384847135242e-08
> > > > 10 KSP preconditioned resid norm 2.480228743414e-11 true resid norm 1.309399823577e-08 ||r(i)||/||b|| 1.777688792258e-09
> > > > 11 KSP preconditioned resid norm 1.728967759950e-12 true resid norm 9.406967045789e-10 ||r(i)||/||b|| 1.277124036931e-10
> > > > 12 KSP preconditioned resid norm 1.075458632828e-13 true resid norm 5.994505136212e-11 ||r(i)||/||b|| 8.138358050689e-12
> > > > Linear solve converged due to CONVERGED_RTOL iterations 12
> > > > KSP Object: 24 MPI processes
> > > > type: gmres
> > > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> > > > happy breakdown tolerance 1e-30
> > > > maximum iterations=40000, initial guess is zero
> > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > left preconditioning
> > > > using PRECONDITIONED norm type for convergence test
> > > > PC Object: 24 MPI processes
> > > > type: hypre
> > > > HYPRE BoomerAMG preconditioning
> > > > Cycle type V
> > > > Maximum number of levels 25
> > > > Maximum number of iterations PER hypre call 1
> > > > Convergence tolerance PER hypre call 0.
> > > > Threshold for strong coupling 0.25
> > > > Interpolation truncation factor 0.
> > > > Interpolation: max elements per row 0
> > > > Number of levels of aggressive coarsening 0
> > > > Number of paths for aggressive coarsening 1
> > > > Maximum row sums 0.9
> > > > Sweeps down 1
> > > > Sweeps up 1
> > > > Sweeps on coarse 1
> > > > Relax down symmetric-SOR/Jacobi
> > > > Relax up symmetric-SOR/Jacobi
> > > > Relax on coarse Gaussian-elimination
> > > > Relax weight (all) 1.
> > > > Outer relax weight (all) 1.
> > > > Using CF-relaxation
> > > > Not using more complex smoothers.
> > > > Measure type local
> > > > Coarsen type Falgout
> > > > Interpolation type classical
> > > > linear system matrix = precond matrix:
> > > > Mat Object: A 24 MPI processes
> > > > type: mpiaij
> > > > rows=497664, cols=497664
> > > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > > total number of mallocs used during MatSetValues calls =0
> > > > has attached null space
> > > > not using I-node (on process 0) routines
> > > > The max singular value of A = 1.000872 in poisson_solver3d_P0_vd
> > > > The min singular value of A = 0.667688 in poisson_solver3d_P0_vd
> > > > The Cond Num of A = 1.499012 in poisson_solver3d_P0_vd
> > > > In poisson_solver3d_pressure(): num_iter = 12, rel_residual = 1.075459e-13
> > > >
> > > > The max value of p0 is 0.03115845493408858
> > > >
> > > > The min value of p0 is -0.07156715468428149
> > > >
> > > > On Sun, Oct 22, 2017 at 12:00 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > >
> > > > > On Oct 21, 2017, at 10:50 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > > >
> > > > > the incompressible NS solver algorithm call PETSc solver at different stage of each time step. The one you were saying "This is good. 12 digit reduction" is after the initial pressure solver, in which usually HYPRE doesn't give a good convergence, so the fall-back solver GMRES will be called after.
> > > >
> > > > Hmm, I don't understand. hypre should do well on a pressure solve. In fact, very well.
> > > > >
> > > > > Barry, you were mentioning that I could have a wrong nullspace. that particular solver is aimed to give an initial pressure profile for 3d incompressible NS simulation using all neumann boundary conditions. could you give some insight how to test if I have a wrong nullspace etc?
> > > >
> > > > -ksp_test_null_space
> > > >
> > > > But if your null space is consistently from all Neumann boundary conditions then it likely is not wrong.
> > > >
> > > > Barry
> > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Sat, Oct 21, 2017 at 11:41 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > > >
> > > > > This is good. You get more than 12 digit reduction in the true residual norm. This is good AMG convergence. Expected when everything goes well.
> > > > >
> > > > > What is different in this case from the previous case that does not converge reasonably?
> > > > >
> > > > > Barry
> > > > >
> > > > >
> > > > > > On Oct 21, 2017, at 9:29 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > > > >
> > > > > > Barry, Please advise what you make of this? this is poisson solver with all neumann BC 3d case Finite difference Scheme was used.
> > > > > > Thanks! I'm in learning mode.
> > > > > >
> > > > > > KSP Object: 24 MPI processes
> > > > > > type: bcgs
> > > > > > maximum iterations=40000, initial guess is zero
> > > > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > > > left preconditioning
> > > > > > using PRECONDITIONED norm type for convergence test
> > > > > > PC Object: 24 MPI processes
> > > > > > type: hypre
> > > > > > HYPRE BoomerAMG preconditioning
> > > > > > Cycle type V
> > > > > > Maximum number of levels 25
> > > > > > Maximum number of iterations PER hypre call 1
> > > > > > Convergence tolerance PER hypre call 0.
> > > > > > Threshold for strong coupling 0.25
> > > > > > Interpolation truncation factor 0.
> > > > > > Interpolation: max elements per row 0
> > > > > > Number of levels of aggressive coarsening 0
> > > > > > Number of paths for aggressive coarsening 1
> > > > > > Maximum row sums 0.9
> > > > > > Sweeps down 1
> > > > > > Sweeps up 1
> > > > > > Sweeps on coarse 1
> > > > > > Relax down symmetric-SOR/Jacobi
> > > > > > Relax up symmetric-SOR/Jacobi
> > > > > > Relax on coarse Gaussian-elimination
> > > > > > Relax weight (all) 1.
> > > > > > Outer relax weight (all) 1.
> > > > > > Using CF-relaxation
> > > > > > Not using more complex smoothers.
> > > > > > Measure type local
> > > > > > Coarsen type Falgout
> > > > > > Interpolation type classical
> > > > > > linear system matrix = precond matrix:
> > > > > > Mat Object: A 24 MPI processes
> > > > > > type: mpiaij
> > > > > > rows=497664, cols=497664
> > > > > > total: nonzeros=3363552, allocated nonzeros=6967296
> > > > > > total number of mallocs used during MatSetValues calls =0
> > > > > > has attached null space
> > > > > > not using I-node (on process 0) routines
> > > > > > 0 KSP preconditioned resid norm 2.697270170623e-02 true resid norm 9.637159071207e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > > > > 1 KSP preconditioned resid norm 4.828857674609e-04 true resid norm 6.294379664645e-01 ||r(i)||/||b|| 6.531364293291e-02
> > > > > > 2 KSP preconditioned resid norm 4.533649595815e-06 true resid norm 1.135508605857e-02 ||r(i)||/||b|| 1.178260727531e-03
> > > > > > 3 KSP preconditioned resid norm 1.131704082606e-07 true resid norm 1.196393029874e-04 ||r(i)||/||b|| 1.241437462051e-05
> > > > > > 4 KSP preconditioned resid norm 3.866281379569e-10 true resid norm 5.121520801846e-07 ||r(i)||/||b|| 5.314347064320e-08
> > > > > > 5 KSP preconditioned resid norm 1.070114785241e-11 true resid norm 1.203693733135e-08 ||r(i)||/||b|| 1.249013038221e-09
> > > > > > 6 KSP preconditioned resid norm 2.578780418765e-14 true resid norm 6.020297525927e-11 ||r(i)||/||b|| 6.246962908306e-12
> > > > > > 7 KSP preconditioned resid norm 8.691764190203e-16 true resid norm 1.866088098154e-12 ||r(i)||/||b|| 1.936346680973e-13
> > > > > > Linear solve converged due to CONVERGED_ATOL iterations 7
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Oct 21, 2017 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > > > >
> > > > > > > On Oct 21, 2017, at 5:47 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > > > > >
> > > > > > > hi, Barry:
> > > > > > > what do you mean absurd by setting tolerance =1e-14?
> > > > > >
> > > > > > Trying to decrease the initial residual norm down by a factor of 1e-14 with an iterative method (or even direct method) is unrealistic, usually unachievable) and almost never necessary. You are requiring || r_n || < 1.e-14 || r_0|| when with double precision numbers you only have roughly 14 decimal digits total to compute with. Round off alone will lead to differences far larger than 1e-14
> > > > > >
> > > > > > If you are using the solver in the context of a nonlinear problem (i.e. inside Newton's method) then 1.e-6 is generally more than plenty to get quadratic convergence of Newton's method.
> > > > > >
> > > > > > If you are solving a linear problem then it is extremely likely that errors due to discretization errors (from finite element method etc) and the model are much much larger than even 1.e-8.
> > > > > >
> > > > > > So, in summary
> > > > > >
> > > > > > 1.e-14 is probably unachievable
> > > > > >
> > > > > > 1.e-14 is almost for sure not needed.
> > > > > >
> > > > > > Barry
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > On Sat, Oct 21, 2017 at 18:42 Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > > > > >
> > > > > > > Run with -ksp_view_mat binary -ksp_view_rhs binary and send the resulting output file called binaryoutput to petsc-maint at mcs.anl.gov
> > > > > > >
> > > > > > > Note you can also use -ksp_type gmres with hypre, unlikely to be a reason to use bcgs
> > > > > > >
> > > > > > > BTW: tolerances: relative=1e-14, is absurd
> > > > > > >
> > > > > > > My guess is your null space is incorrect.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > On Oct 21, 2017, at 4:34 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > > > > > >
> > > > > > > > if this solver doesn't converge. I have a fall-back solution, which uses GMRES solver. this setup is fine with me. I just want to know if HYPRE is a reliable solution for me. Or I will have to go without preconditioner.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > On Sat, Oct 21, 2017 at 5:30 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > > > > > > this is serial run. still dumping output. parallel more or less the same.
> > > > > > > >
> > > > > > > > KSP Object: 1 MPI processes
> > > > > > > > type: bcgs
> > > > > > > > maximum iterations=40000, initial guess is zero
> > > > > > > > tolerances: relative=1e-14, absolute=1e-14, divergence=10000.
> > > > > > > > left preconditioning
> > > > > > > > using PRECONDITIONED norm type for convergence test
> > > > > > > > PC Object: 1 MPI processes
> > > > > > > > type: hypre
> > > > > > > > HYPRE BoomerAMG preconditioning
> > > > > > > > Cycle type V
> > > > > > > > Maximum number of levels 25
> > > > > > > > Maximum number of iterations PER hypre call 1
> > > > > > > > Convergence tolerance PER hypre call 0.
> > > > > > > > Threshold for strong coupling 0.25
> > > > > > > > Interpolation truncation factor 0.
> > > > > > > > Interpolation: max elements per row 0
> > > > > > > > Number of levels of aggressive coarsening 0
> > > > > > > > Number of paths for aggressive coarsening 1
> > > > > > > > Maximum row sums 0.9
> > > > > > > > Sweeps down 1
> > > > > > > > Sweeps up 1
> > > > > > > > Sweeps on coarse 1
> > > > > > > > Relax down symmetric-SOR/Jacobi
> > > > > > > > Relax up symmetric-SOR/Jacobi
> > > > > > > > Relax on coarse Gaussian-elimination
> > > > > > > > Relax weight (all) 1.
> > > > > > > > Outer relax weight (all) 1.
> > > > > > > > Using CF-relaxation
> > > > > > > > Not using more complex smoothers.
> > > > > > > > Measure type local
> > > > > > > > Coarsen type Falgout
> > > > > > > > Interpolation type classical
> > > > > > > > linear system matrix = precond matrix:
> > > > > > > > Mat Object: A 1 MPI processes
> > > > > > > > type: seqaij
> > > > > > > > rows=497664, cols=497664
> > > > > > > > total: nonzeros=3363552, allocated nonzeros=3483648
> > > > > > > > total number of mallocs used during MatSetValues calls =0
> > > > > > > > has attached null space
> > > > > > > > not using I-node routines
> > > > > > > > 0 KSP preconditioned resid norm 1.630377897956e+01 true resid norm 7.365742695123e+00 ||r(i)||/||b|| 1.000000000000e+00
> > > > > > > > 1 KSP preconditioned resid norm 3.815819909205e-02 true resid norm 7.592300567353e-01 ||r(i)||/||b|| 1.030758320187e-01
> > > > > > > > 2 KSP preconditioned resid norm 4.312671277701e-04 true resid norm 2.553060965521e-02 ||r(i)||/||b|| 3.466128360975e-03
> > > > > > > > 3 KSP preconditioned resid norm 3.011875330569e-04 true resid norm 6.836208627386e-04 ||r(i)||/||b|| 9.281085303065e-05
> > > > > > > > 4 KSP preconditioned resid norm 3.011783821295e-04 true resid norm 2.504661140204e-04 ||r(i)||/||b|| 3.400418998972e-05
> > > > > > > > 5 KSP preconditioned resid norm 3.011783818372e-04 true resid norm 2.504053673004e-04 ||r(i)||/||b|| 3.399594279422e-05
> > > > > > > > 6 KSP preconditioned resid norm 3.011783818375e-04 true resid norm 2.503984078890e-04 ||r(i)||/||b|| 3.399499795925e-05
> > > > > > > > 7 KSP preconditioned resid norm 3.011783887442e-04 true resid norm 2.504121501772e-04 ||r(i)||/||b|| 3.399686366224e-05
> > > > > > > > 8 KSP preconditioned resid norm 3.010913654181e-04 true resid norm 2.504150259031e-04 ||r(i)||/||b|| 3.399725408124e-05
> > > > > > > > 9 KSP preconditioned resid norm 3.006520688232e-04 true resid norm 2.504061607382e-04 ||r(i)||/||b|| 3.399605051423e-05
> > > > > > > > 10 KSP preconditioned resid norm 3.007309991942e-04 true resid norm 2.503843638523e-04 ||r(i)||/||b|| 3.399309128978e-05
> > > > > > > > 11 KSP preconditioned resid norm 3.015946168077e-04 true resid norm 2.503644677844e-04 ||r(i)||/||b|| 3.399039012728e-05
> > > > > > > > 12 KSP preconditioned resid norm 2.956643907377e-04 true resid norm 2.503863046509e-04 ||r(i)||/||b|| 3.399335477965e-05
> > > > > > > > 13 KSP preconditioned resid norm 2.997992358936e-04 true resid norm 2.504336903093e-04 ||r(i)||/||b|| 3.399978802886e-05
> > > > > > > > 14 KSP preconditioned resid norm 2.481415420420e-05 true resid norm 2.491591201250e-04 ||r(i)||/||b|| 3.382674774806e-05
> > > > > > > > 15 KSP preconditioned resid norm 2.615494786181e-05 true resid norm 2.490353237273e-04 ||r(i)||/||b|| 3.380994069915e-05
> > > > > > > > 16 KSP preconditioned resid norm 2.645126692130e-05 true resid norm 2.490535523344e-04 ||r(i)||/||b|| 3.381241548111e-05
> > > > > > > > 17 KSP preconditioned resid norm 2.667223026209e-05 true resid norm 2.490482602536e-04 ||r(i)||/||b|| 3.381169700898e-05
> > > > > > > > 18 KSP preconditioned resid norm 2.650813432116e-05 true resid norm 2.490473169014e-04 ||r(i)||/||b|| 3.381156893606e-05
> > > > > > > > 19 KSP preconditioned resid norm 2.613309555449e-05 true resid norm 2.490465633690e-04 ||r(i)||/||b|| 3.381146663375e-05
> > > > > > > > 20 KSP preconditioned resid norm 2.644160446804e-05 true resid norm 2.490532739949e-04 ||r(i)||/||b|| 3.381237769272e-05
> > > > > > > > 21 KSP preconditioned resid norm 2.635987608975e-05 true resid norm 2.490499548926e-04 ||r(i)||/||b|| 3.381192707933e-05
> > > > > > > > 22 KSP preconditioned resid norm 2.640527129095e-05 true resid norm 2.490594066529e-04 ||r(i)||/||b|| 3.381321028466e-05
> > > > > > > > 23 KSP preconditioned resid norm 2.627505117691e-05 true resid norm 2.490550162585e-04 ||r(i)||/||b|| 3.381261422875e-05
> > > > > > > > 24 KSP preconditioned resid norm 2.642659196388e-05 true resid norm 2.490504347640e-04 ||r(i)||/||b|| 3.381199222842e-05
> > > > > > > > 25 KSP preconditioned resid norm 2.659432190695e-05 true resid norm 2.490510775152e-04 ||r(i)||/||b|| 3.381207949065e-05
> > > > > > > > 26 KSP preconditioned resid norm 2.687918062951e-05 true resid norm 2.490518882015e-04 ||r(i)||/||b|| 3.381218955237e-05
> > > > > > > > 27 KSP preconditioned resid norm 2.662909048432e-05 true resid norm 2.490446263285e-04 ||r(i)||/||b|| 3.381120365409e-05
> > > > > > > > 28 KSP preconditioned resid norm 2.085466483199e-05 true resid norm 2.490131612366e-04 ||r(i)||/||b|| 3.380693183886e-05
> > > > > > > > 29 KSP preconditioned resid norm 2.098541330282e-05 true resid norm 2.490126933398e-04 ||r(i)||/||b|| 3.380686831549e-05
> > > > > > > > 30 KSP preconditioned resid norm 2.175345180286e-05 true resid norm 2.490098852429e-04 ||r(i)||/||b|| 3.380648707805e-05
> > > > > > > > 31 KSP preconditioned resid norm 2.182182437676e-05 true resid norm 2.490028301020e-04 ||r(i)||/||b|| 3.380552924648e-05
> > > > > > > > 32 KSP preconditioned resid norm 2.152970404369e-05 true resid norm 2.490089939838e-04 ||r(i)||/||b|| 3.380636607747e-05
> > > > > > > > 33 KSP preconditioned resid norm 2.187932450016e-05 true resid norm 2.490085293931e-04 ||r(i)||/||b|| 3.380630300295e-05
> > > > > > > > 34 KSP preconditioned resid norm 2.207255875067e-05 true resid norm 2.490039036092e-04 ||r(i)||/||b|| 3.380567498971e-05
> > > > > > > > 35 KSP preconditioned resid norm 2.205060279701e-05 true resid norm 2.490101636150e-04 ||r(i)||/||b|| 3.380652487086e-05
> > > > > > > > 36 KSP preconditioned resid norm 2.168654200416e-05 true resid norm 2.490091609876e-04 ||r(i)||/||b|| 3.380638875052e-05
> > > > > > > > 37 KSP preconditioned resid norm 2.164521042361e-05 true resid norm 2.490083143913e-04 ||r(i)||/||b|| 3.380627381352e-05
> > > > > > > > 38 KSP preconditioned resid norm 2.154429063973e-05 true resid norm 2.490075485470e-04 ||r(i)||/||b|| 3.380616983972e-05
> > > > > > > > 39 KSP preconditioned resid norm 2.165962086228e-05 true resid norm 2.490099695056e-04 ||r(i)||/||b|| 3.380649851786e-05
> > > > > > > > 40 KSP preconditioned resid norm 2.153877616091e-05 true resid norm 2.490090652619e-04 ||r(i)||/||b|| 3.380637575444e-05
> > > > > > > > 41 KSP preconditioned resid norm 2.347651187611e-05 true resid norm 2.490233544624e-04 ||r(i)||/||b|| 3.380831570825e-05
> > > > > > > > 42 KSP preconditioned resid norm 2.352860162514e-05 true resid norm 2.490191394202e-04 ||r(i)||/||b|| 3.380774345879e-05
> > > > > > > > 43 KSP preconditioned resid norm 2.312377506928e-05 true resid norm 2.490209491359e-04 ||r(i)||/||b|| 3.380798915237e-05
> > > > > > > > 44 KSP preconditioned resid norm 2.295770973533e-05 true resid norm 2.490178136759e-04 ||r(i)||/||b|| 3.380756347093e-05
> > > > > > > > 45 KSP preconditioned resid norm 2.833646456041e-05 true resid norm 2.489991602651e-04 ||r(i)||/||b|| 3.380503101608e-05
> > > > > > > > 46 KSP preconditioned resid norm 2.760296424494e-05 true resid norm 2.490104320666e-04 ||r(i)||/||b|| 3.380656131682e-05
> > > > > > > > 47 KSP preconditioned resid norm 2.451504295239e-05 true resid norm 2.490241388672e-04 ||r(i)||/||b|| 3.380842220189e-05
> > > > > > > > 48 KSP preconditioned resid norm 2.512391514098e-05 true resid norm 2.490245923753e-04 ||r(i)||/||b|| 3.380848377180e-05
> > > > > > > > 49 KSP preconditioned resid norm 2.483419450528e-05 true resid norm 2.490273364402e-04 ||r(i)||/||b|| 3.380885631602e-05
> > > > > > > > 50 KSP preconditioned resid norm 2.507460538466e-05 true resid norm 2.490309488780e-04 ||r(i)||/||b|| 3.380934675371e-05
> > > > > > > > 51 KSP preconditioned resid norm 2.499708772881e-05 true resid norm 2.490300908170e-04 ||r(i)||/||b|| 3.380923026022e-05
> > > > > > > > 52 KSP preconditioned resid norm 1.059778259446e-05 true resid norm 2.489352833521e-04 ||r(i)||/||b|| 3.379635885420e-05
> > > > > > > > 53 KSP preconditioned resid norm 1.074975117060e-05 true resid norm 2.489294722901e-04 ||r(i)||/||b|| 3.379556992330e-05
> > > > > > > > 54 KSP preconditioned resid norm 1.095242219559e-05 true resid norm 2.489295454212e-04 ||r(i)||/||b|| 3.379557985184e-05
> > > > > > > > 55 KSP preconditioned resid norm 8.359999674720e-06 true resid norm 2.489673581944e-04 ||r(i)||/||b|| 3.380071345137e-05
> > > > > > > > 56 KSP preconditioned resid norm 8.368232998470e-06 true resid norm 2.489700421343e-04 ||r(i)||/||b|| 3.380107783281e-05
> > > > > > > > 57 KSP preconditioned resid norm 8.443378041101e-06 true resid norm 2.489702900875e-04 ||r(i)||/||b|| 3.380111149584e-05
> > > > > > > > 58 KSP preconditioned resid norm 8.647159584302e-06 true resid norm 2.489640805831e-04 ||r(i)||/||b|| 3.380026847095e-05
> > > > > > > > 59 KSP preconditioned resid norm 1.024742790737e-05 true resid norm 2.489447846660e-04 ||r(i)||/||b|| 3.379764878711e-05
> > > > > > > > 60 KSP preconditioned resid norm 1.033394118910e-05 true resid norm 2.489441404923e-04 ||r(i)||/||b|| 3.379756133175e-05
> > > > > > > > 61 KSP preconditioned resid norm 1.030066336446e-05 true resid norm 2.489399918556e-04 ||r(i)||/||b|| 3.379699809776e-05
> > > > > > > > 62 KSP preconditioned resid norm 1.029956398963e-05 true resid norm 2.489445295139e-04 ||r(i)||/||b|| 3.379761414674e-05
> > > > > > > > 63 KSP preconditioned resid norm 1.028190129002e-05 true resid norm 2.489456200527e-04 ||r(i)||/||b|| 3.379776220225e-05
> > > > > > > > 64 KSP preconditioned resid norm 9.878799185773e-06 true resid norm 2.489488742330e-04 ||r(i)||/||b|| 3.379820400160e-05
> > > > > > > > 65 KSP preconditioned resid norm 9.917711104174e-06 true resid norm 2.489478066593e-04 ||r(i)||/||b|| 3.379805906391e-05
> > > > > > > > 66 KSP preconditioned resid norm 1.003572019576e-05 true resid norm 2.489441995703e-04 ||r(i)||/||b|| 3.379756935240e-05
> > > > > > > > 67 KSP preconditioned resid norm 9.924487278236e-06 true resid norm 2.489475403451e-04 ||r(i)||/||b|| 3.379802290812e-05
> > > > > > > > 68 KSP preconditioned resid norm 9.804213483359e-06 true resid norm 2.489457781760e-04 ||r(i)||/||b|| 3.379778366964e-05
> > > > > > > > 69 KSP preconditioned resid norm 9.748922705476e-06 true resid norm 2.489408473578e-04 ||r(i)||/||b|| 3.379711424383e-05
> > > > > > > > 70 KSP preconditioned resid norm 9.886044523689e-06 true resid norm 2.489514438395e-04 ||r(i)||/||b|| 3.379855286071e-05
> > > > > > > > 71 KSP preconditioned resid norm 1.083888478689e-05 true resid norm 2.489420898851e-04 ||r(i)||/||b|| 3.379728293386e-05
> > > > > > > > 72 KSP preconditioned resid norm 1.106561823757e-05 true resid norm 2.489364778104e-04 ||r(i)||/||b|| 3.379652101821e-05
> > > > > > > > 73 KSP preconditioned resid norm 1.132091515426e-05 true resid norm 2.489456804535e-04 ||r(i)||/||b|| 3.379777040248e-05
> > > > > > > > 74 KSP preconditioned resid norm 1.330905328963e-05 true resid norm 2.489317458981e-04 ||r(i)||/||b|| 3.379587859660e-05
> > > > > > > > 75 KSP preconditioned resid norm 1.305555302619e-05 true resid norm 2.489320939810e-04 ||r(i)||/||b|| 3.379592585359e-05
> > > > > > > > 76 KSP preconditioned resid norm 1.308083397399e-05 true resid norm 2.489299951581e-04 ||r(i)||/||b|| 3.379564090977e-05
> > > > > > > > 77 KSP preconditioned resid norm 1.320098861853e-05 true resid norm 2.489323669317e-04 ||r(i)||/||b|| 3.379596291036e-05
> > > > > > > > 78 KSP preconditioned resid norm 1.300160788274e-05 true resid norm 2.489306393356e-04 ||r(i)||/||b|| 3.379572836564e-05
> > > > > > > > 79 KSP preconditioned resid norm 1.317651537793e-05 true resid norm 2.489381364970e-04 ||r(i)||/||b|| 3.379674620752e-05
> > > > > > > > 80 KSP preconditioned resid norm 1.309769805765e-05 true resid norm 2.489285056062e-04 ||r(i)||/||b|| 3.379543868279e-05
> > > > > > > > 81 KSP preconditioned resid norm 1.293686496271e-05 true resid norm 2.489347818072e-04 ||r(i)||/||b|| 3.379629076264e-05
> > > > > > > > 82 KSP preconditioned resid norm 1.311788285799e-05 true resid norm 2.489320040215e-04 ||r(i)||/||b|| 3.379591364037e-05
> > > > > > > > 83 KSP preconditioned resid norm 1.313667378798e-05 true resid norm 2.489329437217e-04 ||r(i)||/||b|| 3.379604121748e-05
> > > > > > > > 84 KSP preconditioned resid norm 1.416138205017e-05 true resid norm 2.489266908838e-04 ||r(i)||/||b|| 3.379519230948e-05
> > > > > > > > 85 KSP preconditioned resid norm 1.452253464774e-05 true resid norm 2.489285688375e-04 ||r(i)||/||b|| 3.379544726729e-05
> > > > > > > > 86 KSP preconditioned resid norm 1.426709413370e-05 true resid norm 2.489362313402e-04 ||r(i)||/||b|| 3.379648755651e-05
> > > > > > > > 87 KSP preconditioned resid norm 1.427480849552e-05 true resid norm 2.489378183000e-04 ||r(i)||/||b|| 3.379670300795e-05
> > > > > > > > 88 KSP preconditioned resid norm 1.413870980147e-05 true resid norm 2.489325756118e-04 ||r(i)||/||b|| 3.379599124153e-05
> > > > > > > > 89 KSP preconditioned resid norm 1.353259857657e-05 true resid norm 2.489318968308e-04 ||r(i)||/||b|| 3.379589908776e-05
> > > > > > > > 90 KSP preconditioned resid norm 1.347676448611e-05 true resid norm 2.489332074417e-04 ||r(i)||/||b|| 3.379607702106e-05
> > > > > > > > 91 KSP preconditioned resid norm 1.362825902909e-05 true resid norm 2.489344974971e-04 ||r(i)||/||b|| 3.379625216367e-05
> > > > > > > > 92 KSP preconditioned resid norm 1.346280901052e-05 true resid norm 2.489302570131e-04 ||r(i)||/||b|| 3.379567646016e-05
> > > > > > > > 93 KSP preconditioned resid norm 1.328052169696e-05 true resid norm 2.489346601224e-04 ||r(i)||/||b|| 3.379627424228e-05
> > > > > > > > 94 KSP preconditioned resid norm 1.554682082515e-05 true resid norm 2.489309078759e-04 ||r(i)||/||b|| 3.379576482365e-05
> > > > > > > > 95 KSP preconditioned resid norm 1.557128675775e-05 true resid norm 2.489317143582e-04 ||r(i)||/||b|| 3.379587431462e-05
> > > > > > > > 96 KSP preconditioned resid norm 1.542571813923e-05 true resid norm 2.489319910303e-04 ||r(i)||/||b|| 3.379591187663e-05
> > > > > > > > 97 KSP preconditioned resid norm 1.570516684444e-05 true resid norm 2.489321980894e-04 ||r(i)||/||b|| 3.379593998772e-05
> > > > > > > > 98 KSP preconditioned resid norm 1.600431789899e-05 true resid norm 2.489297450311e-04 ||r(i)||/||b|| 3.379560695162e-05
> > > > > > > > 99 KSP preconditioned resid norm 1.587495554658e-05 true resid norm 2.489339000570e-04 ||r(i)||/||b|| 3.379617105303e-05
> > > > > > > > 100 KSP preconditioned resid norm 1.621163002878e-05 true resid norm 2.489299953360e-04 ||r(i)||/||b|| 3.379564093392e-05
> > > > > > > > 101 KSP preconditioned resid norm 1.627060872574e-05 true resid norm 2.489301570161e-04 ||r(i)||/||b|| 3.379566288419e-05
> > > > > > > > 102 KSP preconditioned resid norm 1.622931647243e-05 true resid norm 2.489277930910e-04 ||r(i)||/||b|| 3.379534194913e-05
> > > > > > > > 103 KSP preconditioned resid norm 1.612544300282e-05 true resid norm 2.489317483299e-04 ||r(i)||/||b|| 3.379587892674e-05
> > > > > > > > 104 KSP preconditioned resid norm 1.880131646630e-05 true resid norm 2.489335862583e-04 ||r(i)||/||b|| 3.379612845059e-05
> > > > > > > > 105 KSP preconditioned resid norm 1.880563295793e-05 true resid norm 2.489365017923e-04 ||r(i)||/||b|| 3.379652427408e-05
> > > > > > > > 106 KSP preconditioned resid norm 1.860619184027e-05 true resid norm 2.489362382373e-04 ||r(i)||/||b|| 3.379648849288e-05
> > > > > > > > 107 KSP preconditioned resid norm 1.877134148719e-05 true resid norm 2.489425484523e-04 ||r(i)||/||b|| 3.379734519061e-05
> > > > > > > > 108 KSP preconditioned resid norm 1.914810713538e-05 true resid norm 2.489347415573e-04 ||r(i)||/||b|| 3.379628529818e-05
> > > > > > > > 109 KSP preconditioned resid norm 1.220673255622e-05 true resid norm 2.490196186357e-04 ||r(i)||/||b|| 3.380780851884e-05
> > > > > > > > 110 KSP preconditioned resid norm 1.215819132910e-05 true resid norm 2.490233518072e-04 ||r(i)||/||b|| 3.380831534776e-05
> > > > > > > > 111 KSP preconditioned resid norm 1.196565427400e-05 true resid norm 2.490279438906e-04 ||r(i)||/||b|| 3.380893878570e-05
> > > > > > > > 112 KSP preconditioned resid norm 1.171748185197e-05 true resid norm 2.490242578983e-04 ||r(i)||/||b|| 3.380843836198e-05
> > > > > > > > 113 KSP preconditioned resid norm 1.162855824118e-05 true resid norm 2.490229018536e-04 ||r(i)||/||b|| 3.380825426043e-05
> > > > > > > > 114 KSP preconditioned resid norm 1.175594685689e-05 true resid norm 2.490274328440e-04 ||r(i)||/||b|| 3.380886940415e-05
> > > > > > > > 115 KSP preconditioned resid norm 1.167979454122e-05 true resid norm 2.490271161036e-04 ||r(i)||/||b|| 3.380882640232e-05
> > > > > > > > 116 KSP preconditioned resid norm 1.181010893019e-05 true resid norm 2.490235420657e-04 ||r(i)||/||b|| 3.380834117795e-05
> > > > > > > > 117 KSP preconditioned resid norm 1.175206638194e-05 true resid norm 2.490263165345e-04 ||r(i)||/||b|| 3.380871784992e-05
> > > > > > > > 118 KSP preconditioned resid norm 1.183804125791e-05 true resid norm 2.490221353083e-04 ||r(i)||/||b|| 3.380815019145e-05
> > > > > > > > 119 KSP preconditioned resid norm 1.186426973727e-05 true resid norm 2.490227115336e-04 ||r(i)||/||b|| 3.380822842189e-05
> > > > > > > > 120 KSP preconditioned resid norm 1.181986776689e-05 true resid norm 2.490257884230e-04 ||r(i)||/||b|| 3.380864615159e-05
> > > > > > > > 121 KSP preconditioned resid norm 1.131443277370e-05 true resid norm 2.490259110230e-04 ||r(i)||/||b|| 3.380866279620e-05
> > > > > > > > 122 KSP preconditioned resid norm 1.114920075859e-05 true resid norm 2.490249829382e-04 ||r(i)||/||b|| 3.380853679603e-05
> > > > > > > > 123 KSP preconditioned resid norm 1.082073321672e-05 true resid norm 2.490314084868e-04 ||r(i)||/||b|| 3.380940915187e-05
> > > > > > > > 124 KSP preconditioned resid norm 3.307785860990e-06 true resid norm 2.490613501549e-04 ||r(i)||/||b|| 3.381347414155e-05
> > > > > > > > 125 KSP preconditioned resid norm 3.287051720572e-06 true resid norm 2.490584648195e-04 ||r(i)||/||b|| 3.381308241794e-05
> > > > > > > > 126 KSP preconditioned resid norm 3.286797046069e-06 true resid norm 2.490654396386e-04 ||r(i)||/||b|| 3.381402934473e-05
> > > > > > > > 127 KSP preconditioned resid norm 3.311592899411e-06 true resid norm 2.490627973588e-04 ||r(i)||/||b|| 3.381367061922e-05
> > > > > > > > 128 KSP preconditioned resid norm 3.560993694635e-06 true resid norm 2.490571732816e-04 ||r(i)||/||b|| 3.381290707406e-05
> > > > > > > > 129 KSP preconditioned resid norm 3.411994617661e-06 true resid norm 2.490652122141e-04 ||r(i)||/||b|| 3.381399846875e-05
> > > > > > > > 130 KSP preconditioned resid norm 3.412383310721e-06 true resid norm 2.490633454022e-04 ||r(i)||/||b|| 3.381374502359e-05
> > > > > > > > 131 KSP preconditioned resid norm 3.288320044878e-06 true resid norm 2.490639470096e-04 ||r(i)||/||b|| 3.381382669999e-05
> > > > > > > > 132 KSP preconditioned resid norm 3.273215756565e-06 true resid norm 2.490640390847e-04 ||r(i)||/||b|| 3.381383920043e-05
> > > > > > > > 133 KSP preconditioned resid norm 3.236969051459e-06 true resid norm 2.490678216102e-04 ||r(i)||/||b|| 3.381435272985e-05
> > > > > > > > 134 KSP preconditioned resid norm 3.203260913942e-06 true resid norm 2.490640965346e-04 ||r(i)||/||b|| 3.381384700005e-05
> > > > > > > > 135 KSP preconditioned resid norm 3.224117152353e-06 true resid norm 2.490655026376e-04 ||r(i)||/||b|| 3.381403789770e-05
> > > > > > > > 136 KSP preconditioned resid norm 3.221577997984e-06 true resid norm 2.490684737611e-04 ||r(i)||/||b|| 3.381444126823e-05
> > > > > > > > 137 KSP preconditioned resid norm 3.195936222128e-06 true resid norm 2.490673982333e-04 ||r(i)||/||b|| 3.381429525066e-05
> > > > > > > > 138 KSP preconditioned resid norm 3.207528137426e-06 true resid norm 2.490641247196e-04 ||r(i)||/||b|| 3.381385082655e-05
> > > > > > > > 139 KSP preconditioned resid norm 3.240134271963e-06 true resid norm 2.490615861251e-04 ||r(i)||/||b|| 3.381350617773e-05
> > > > > > > > 140 KSP preconditioned resid norm 2.698833607230e-06 true resid norm 2.490638954889e-04 ||r(i)||/||b|| 3.381381970535e-05
> > > > > > > > 141 KSP preconditioned resid norm 2.599151209137e-06 true resid norm 2.490657106698e-04 ||r(i)||/||b|| 3.381406614091e-05
> > > > > > > > 142 KSP preconditioned resid norm 2.633939920994e-06 true resid norm 2.490707754695e-04 ||r(i)||/||b|| 3.381475375652e-05
> > > > > > > > 143 KSP preconditioned resid norm 2.519609221376e-06 true resid norm 2.490639100480e-04 ||r(i)||/||b|| 3.381382168195e-05
> > > > > > > > 144 KSP preconditioned resid norm 3.768526937684e-06 true resid norm 2.490654096698e-04 ||r(i)||/||b|| 3.381402527606e-05
> > > > > > > > 145 KSP preconditioned resid norm 3.707841943289e-06 true resid norm 2.490630207923e-04 ||r(i)||/||b|| 3.381370095336e-05
> > > > > > > > 146 KSP preconditioned resid norm 3.698827503486e-06 true resid norm 2.490646071561e-04 ||r(i)||/||b|| 3.381391632387e-05
> > > > > > > > 147 KSP preconditioned resid norm 3.642747039615e-06 true resid norm 2.490610990161e-04 ||r(i)||/||b|| 3.381344004604e-05
> > > > > > > > 148 KSP preconditioned resid norm 3.613100087842e-06 true resid norm 2.490617159023e-04 ||r(i)||/||b|| 3.381352379676e-05
> > > > > > > > 149 KSP preconditioned resid norm 3.637646399299e-06 true resid norm 2.490648063023e-04 ||r(i)||/||b|| 3.381394336069e-05
> > > > > > > > 150 KSP preconditioned resid norm 3.640235367864e-06 true resid norm 2.490648516718e-04 ||r(i)||/||b|| 3.381394952022e-05
> > > > > > > > 151 KSP preconditioned resid norm 3.724708848977e-06 true resid norm 2.490622201040e-04 ||r(i)||/||b|| 3.381359224901e-05
> > > > > > > > 152 KSP preconditioned resid norm 3.665185002770e-06 true resid norm 2.490664302790e-04 ||r(i)||/||b|| 3.381416383766e-05
> > > > > > > > 153 KSP preconditioned resid norm 3.348992579120e-06 true resid norm 2.490655722697e-04 ||r(i)||/||b|| 3.381404735121e-05
> > > > > > > > 154 KSP preconditioned resid norm 3.309431137943e-06 true resid norm 2.490727563300e-04 ||r(i)||/||b|| 3.381502268535e-05
> > > > > > > > 155 KSP preconditioned resid norm 3.299031245428e-06 true resid norm 2.490688392843e-04 ||r(i)||/||b|| 3.381449089298e-05
> > > > > > > > 156 KSP preconditioned resid norm 3.297127463503e-06 true resid norm 2.490642207769e-04 ||r(i)||/||b|| 3.381386386763e-05
> > > > > > > > 157 KSP preconditioned resid norm 3.297370198641e-06 true resid norm 2.490666651723e-04 ||r(i)||/||b|| 3.381419572764e-05
> > > > > > > > 158 KSP preconditioned resid norm 3.290873165210e-06 true resid norm 2.490679189538e-04 ||r(i)||/||b|| 3.381436594557e-05
> > > > > > > > 159 KSP preconditioned resid norm 3.346705292419e-06 true resid norm 2.490617329776e-04 ||r(i)||/||b|| 3.381352611496e-05
> > > > > > > > 160 KSP preconditioned resid norm 3.429583550890e-06 true resid norm 2.490675116236e-04 ||r(i)||/||b|| 3.381431064494e-05
> > > > > > > > 161 KSP preconditioned resid norm 3.425238504679e-06 true resid norm 2.490648199058e-04 ||r(i)||/||b|| 3.381394520756e-05
> > > > > > > > 162 KSP preconditioned resid norm 3.423484857849e-06 true resid norm 2.490723208298e-04 ||r(i)||/||b|| 3.381496356025e-05
> > > > > > > > 163 KSP preconditioned resid norm 3.383655922943e-06 true resid norm 2.490659981249e-04 ||r(i)||/||b|| 3.381410516686e-05
> > > > > > > > 164 KSP preconditioned resid norm 3.477197358452e-06 true resid norm 2.490665979073e-04 ||r(i)||/||b|| 3.381418659549e-05
> > > > > > > > 165 KSP preconditioned resid norm 3.454672202601e-06 true resid norm 2.490651358644e-04 ||r(i)||/||b|| 3.381398810323e-05
> > > > > > > > 166 KSP preconditioned resid norm 3.399075522566e-06 true resid norm 2.490678159511e-04 ||r(i)||/||b|| 3.381435196154e-05
> > > > > > > > 167 KSP preconditioned resid norm 3.305455787400e-06 true resid norm 2.490651924523e-04 ||r(i)||/||b|| 3.381399578581e-05
> > > > > > > > 168 KSP preconditioned resid norm 3.368445533284e-06 true resid norm 2.490688061735e-04 ||r(i)||/||b|| 3.381448639774e-05
> > > > > > > > 169 KSP preconditioned resid norm 2.981519724814e-06 true resid norm 2.490676378334e-04 ||r(i)||/||b|| 3.381432777964e-05
> > > > > > > > 170 KSP preconditioned resid norm 3.034423065539e-06 true resid norm 2.490694458885e-04 ||r(i)||/||b|| 3.381457324777e-05
> > > > > > > > 171 KSP preconditioned resid norm 2.885972780503e-06 true resid norm 2.490688033729e-04 ||r(i)||/||b|| 3.381448601752e-05
> > > > > > > > 172 KSP preconditioned resid norm 2.892491075033e-06 true resid norm 2.490692993765e-04 ||r(i)||/||b|| 3.381455335678e-05
> > > > > > > > 173 KSP preconditioned resid norm 2.921316177611e-06 true resid norm 2.490697629787e-04 ||r(i)||/||b|| 3.381461629709e-05
> > > > > > > > 174 KSP preconditioned resid norm 2.999889222269e-06 true resid norm 2.490707272626e-04 ||r(i)||/||b|| 3.381474721178e-05
> > > > > > > > 175 KSP preconditioned resid norm 2.975590207575e-06 true resid norm 2.490685439925e-04 ||r(i)||/||b|| 3.381445080310e-05
> > > > > > > > 176 KSP preconditioned resid norm 2.983065843597e-06 true resid norm 2.490701883671e-04 ||r(i)||/||b|| 3.381467404937e-05
> > > > > > > > 177 KSP preconditioned resid norm 2.965959610245e-06 true resid norm 2.490711538630e-04 ||r(i)||/||b|| 3.381480512861e-05
> > > > > > > > 178 KSP preconditioned resid norm 3.005389788827e-06 true resid norm 2.490751808095e-04 ||r(i)||/||b|| 3.381535184150e-05
> > > > > > > > 179 KSP preconditioned resid norm 2.956581668772e-06 true resid norm 2.490653125636e-04 ||r(i)||/||b|| 3.381401209257e-05
> > > > > > > > 180 KSP preconditioned resid norm 2.937498883661e-06 true resid norm 2.490666056653e-04 ||r(i)||/||b|| 3.381418764874e-05
> > > > > > > > 181 KSP preconditioned resid norm 2.913227475431e-06 true resid norm 2.490682436979e-04 ||r(i)||/||b|| 3.381441003402e-05
> > > > > > > > 182 KSP preconditioned resid norm 3.048172862254e-06 true resid norm 2.490719669872e-04 ||r(i)||/||b|| 3.381491552130e-05
> > > > > > > > 183 KSP preconditioned resid norm 3.023868104933e-06 true resid norm 2.490648745555e-04 ||r(i)||/||b|| 3.381395262699e-05
> > > > > > > > 184 KSP preconditioned resid norm 2.985947506400e-06 true resid norm 2.490638818852e-04 ||r(i)||/||b|| 3.381381785846e-05
> > > > > > > > 185 KSP preconditioned resid norm 2.840032055776e-06 true resid norm 2.490701112392e-04 ||r(i)||/||b|| 3.381466357820e-05
> > > > > > > > 186 KSP preconditioned resid norm 2.229279683815e-06 true resid norm 2.490609220680e-04 ||r(i)||/||b|| 3.381341602292e-05
> > > > > > > > 187 KSP preconditioned resid norm 2.441513276379e-06 true resid norm 2.490674056899e-04 ||r(i)||/||b|| 3.381429626300e-05
> > > > > > > > 188 KSP preconditioned resid norm 2.467046864016e-06 true resid norm 2.490691622632e-04 ||r(i)||/||b|| 3.381453474178e-05
> > > > > > > > 189 KSP preconditioned resid norm 2.482124586361e-06 true resid norm 2.490664992339e-04 ||r(i)||/||b|| 3.381417319923e-05
> > > > > > > > 190 KSP preconditioned resid norm 2.470564926502e-06 true resid norm 2.490617019713e-04 ||r(i)||/||b|| 3.381352190543e-05
> > > > > > > > 191 KSP preconditioned resid norm 2.457947086578e-06 true resid norm 2.490628644250e-04 ||r(i)||/||b|| 3.381367972437e-05
> > > > > > > > 192 KSP preconditioned resid norm 2.469444741724e-06 true resid norm 2.490639416335e-04 ||r(i)||/||b|| 3.381382597011e-05
> > > > > > > > 193 KSP preconditioned resid norm 2.469951525219e-06 true resid norm 2.490599769764e-04 ||r(i)||/||b|| 3.381328771385e-05
> > > > > > > > 194 KSP preconditioned resid norm 2.467486786643e-06 true resid norm 2.490630178622e-04 ||r(i)||/||b|| 3.381370055556e-05
> > > > > > > > 195 KSP preconditioned resid norm 2.409684391404e-06 true resid norm 2.490640302606e-04 ||r(i)||/||b|| 3.381383800245e-05
> > > > > > > > 196 KSP preconditioned resid norm 2.456046691135e-06 true resid norm 2.490637730235e-04 ||r(i)||/||b|| 3.381380307900e-05
> > > > > > > > 197 KSP preconditioned resid norm 2.300015653805e-06 true resid norm 2.490615406913e-04 ||r(i)||/||b|| 3.381350000947e-05
> > > > > > > > 198 KSP preconditioned resid norm 2.238328275301e-06 true resid norm 2.490647641246e-04 ||r(i)||/||b|| 3.381393763449e-05
> > > > > > > > 199 KSP preconditioned resid norm 2.317293820319e-06 true resid norm 2.490641611282e-04 ||r(i)||/||b|| 3.381385576951e-05
> > > > > > > > 200 KSP preconditioned resid norm 2.359590971314e-06 true resid norm 2.490685242974e-04 ||r(i)||/||b|| 3.381444812922e-05
> > > > > > > > 201 KSP preconditioned resid norm 2.311199691596e-06 true resid norm 2.490656791753e-04 ||r(i)||/||b|| 3.381406186510e-05
> > > > > > > > 202 KSP preconditioned resid norm 2.328772904196e-06 true resid norm 2.490651045523e-04 ||r(i)||/||b|| 3.381398385220e-05
> > > > > > > > 203 KSP preconditioned resid norm 2.332731604717e-06 true resid norm 2.490649960574e-04 ||r(i)||/||b|| 3.381396912253e-05
> > > > > > > > 204 KSP preconditioned resid norm 2.357629383490e-06 true resid norm 2.490686317727e-04 ||r(i)||/||b|| 3.381446272046e-05
> > > > > > > > 205 KSP preconditioned resid norm 2.374856180299e-06 true resid norm 2.490645897176e-04 ||r(i)||/||b|| 3.381391395637e-05
> > > > > > > > 206 KSP preconditioned resid norm 2.340395514404e-06 true resid norm 2.490618341127e-04 ||r(i)||/||b|| 3.381353984542e-05
> > > > > > > > 207 KSP preconditioned resid norm 2.314963680954e-06 true resid norm 2.490676153984e-04 ||r(i)||/||b|| 3.381432473379e-05
> > > > > > > > 208 KSP preconditioned resid norm 2.448070953106e-06 true resid norm 2.490644606776e-04 ||r(i)||/||b|| 3.381389643743e-05
> > > > > > > > 209 KSP preconditioned resid norm 2.428805110632e-06 true resid norm 2.490635817597e-04 ||r(i)||/||b|| 3.381377711234e-05
> > > > > > > > 210 KSP preconditioned resid norm 2.537929937808e-06 true resid norm 2.490680589404e-04 ||r(i)||/||b|| 3.381438495066e-05
> > > > > > > > 211 KSP preconditioned resid norm 2.515909029682e-06 true resid norm 2.490687803038e-04 ||r(i)||/||b|| 3.381448288557e-05
> > > > > > > > 212 KSP preconditioned resid norm 2.497907513266e-06 true resid norm 2.490618016885e-04 ||r(i)||/||b|| 3.381353544340e-05
> > > > > > > > 213 KSP preconditioned resid norm 1.783501869502e-06 true resid norm 2.490632647470e-04 ||r(i)||/||b|| 3.381373407354e-05
> > > > > > > > 214 KSP preconditioned resid norm 1.767420653144e-06 true resid norm 2.490685569328e-04 ||r(i)||/||b|| 3.381445255992e-05
> > > > > > > > 215 KSP preconditioned resid norm 1.854926068272e-06 true resid norm 2.490609365464e-04 ||r(i)||/||b|| 3.381341798856e-05
> > > > > > > > 216 KSP preconditioned resid norm 1.818308539774e-06 true resid norm 2.490639142283e-04 ||r(i)||/||b|| 3.381382224948e-05
> > > > > > > > 217 KSP preconditioned resid norm 1.809431578070e-06 true resid norm 2.490605125049e-04 ||r(i)||/||b|| 3.381336041915e-05
> > > > > > > > 218 KSP preconditioned resid norm 1.789862735999e-06 true resid norm 2.490564024901e-04 ||r(i)||/||b|| 3.381280242859e-05
> > > > > > > > 219 KSP preconditioned resid norm 1.769239890163e-06 true resid norm 2.490647825316e-04 ||r(i)||/||b|| 3.381394013349e-05
> > > > > > > > 220 KSP preconditioned resid norm 1.780760773109e-06 true resid norm 2.490622606663e-04 ||r(i)||/||b|| 3.381359775589e-05
> > > > > > > > 221 KSP preconditioned resid norm 5.009024913368e-07 true resid norm 2.490659101637e-04 ||r(i)||/||b|| 3.381409322492e-05
> > > > > > > > 222 KSP preconditioned resid norm 4.974450322799e-07 true resid norm 2.490714287402e-04 ||r(i)||/||b|| 3.381484244693e-05
> > > > > > > > 223 KSP preconditioned resid norm 4.938819481519e-07 true resid norm 2.490665661715e-04 ||r(i)||/||b|| 3.381418228693e-05
> > > > > > > > 224 KSP preconditioned resid norm 4.973231831266e-07 true resid norm 2.490725000995e-04 ||r(i)||/||b|| 3.381498789855e-05
> > > > > > > > 225 KSP preconditioned resid norm 5.086864036771e-07 true resid norm 2.490664132954e-04 ||r(i)||/||b|| 3.381416153192e-05
> > > > > > > > 226 KSP preconditioned resid norm 5.046954570561e-07 true resid norm 2.490698772594e-04 ||r(i)||/||b|| 3.381463181226e-05
> > > > > > > > 227 KSP preconditioned resid norm 5.086852920874e-07 true resid norm 2.490703544723e-04 ||r(i)||/||b|| 3.381469660041e-05
> > > > > > > > 228 KSP preconditioned resid norm 5.182381756169e-07 true resid norm 2.490665200032e-04 ||r(i)||/||b|| 3.381417601896e-05
> > > > > > > > 229 KSP preconditioned resid norm 5.261455182896e-07 true resid norm 2.490697169472e-04 ||r(i)||/||b|| 3.381461004770e-05
> > > > > > > > 230 KSP preconditioned resid norm 5.265262522400e-07 true resid norm 2.490726890541e-04 ||r(i)||/||b|| 3.381501355172e-05
> > > > > > > > 231 KSP preconditioned resid norm 5.220652263946e-07 true resid norm 2.490689325236e-04 ||r(i)||/||b|| 3.381450355149e-05
> > > > > > > > 232 KSP preconditioned resid norm 5.256466259888e-07 true resid norm 2.490694033989e-04 ||r(i)||/||b|| 3.381456747923e-05
> > > > > > > > 233 KSP preconditioned resid norm 5.443022648374e-07 true resid norm 2.490650183144e-04 ||r(i)||/||b|| 3.381397214423e-05
> > > > > > > > 234 KSP preconditioned resid norm 5.562619006436e-07 true resid norm 2.490764576883e-04 ||r(i)||/||b|| 3.381552519520e-05
> > > > > > > > 235 KSP preconditioned resid norm 5.998148629545e-07 true resid norm 2.490714032716e-04 ||r(i)||/||b|| 3.381483898922e-05
> > > > > > > > 236 KSP preconditioned resid norm 6.498977322955e-07 true resid norm 2.490650270144e-04 ||r(i)||/||b|| 3.381397332537e-05
> > > > > > > > 237 KSP preconditioned resid norm 6.503686003429e-07 true resid norm 2.490706976108e-04 ||r(i)||/||b|| 3.381474318615e-05
> > > > > > > > 238 KSP preconditioned resid norm 6.566719023119e-07 true resid norm 2.490664107559e-04 ||r(i)||/||b|| 3.381416118714e-05
> > > > > > > > 239 KSP preconditioned resid norm 6.549737473208e-07 true resid norm 2.490721547909e-04 ||r(i)||/||b|| 3.381494101821e-05
> > > > > > > > 240 KSP preconditioned resid norm 6.616898981418e-07 true resid norm 2.490679659838e-04 ||r(i)||/||b|| 3.381437233053e-05
> > > > > > > > 241 KSP preconditioned resid norm 6.829917691021e-07 true resid norm 2.490728328614e-04 ||r(i)||/||b|| 3.381503307553e-05
> > > > > > > > 242 KSP preconditioned resid norm 7.030239869389e-07 true resid norm 2.490706345115e-04 ||r(i)||/||b|| 3.381473461955e-05
> > > > > > > > 243 KSP preconditioned resid norm 7.018435683340e-07 true resid norm 2.490650978460e-04 ||r(i)||/||b|| 3.381398294172e-05
> > > > > > > > 244 KSP preconditioned resid norm 7.058047080376e-07 true resid norm 2.490685975642e-04 ||r(i)||/||b|| 3.381445807618e-05
> > > > > > > > 245 KSP preconditioned resid norm 6.896300385099e-07 true resid norm 2.490708566380e-04 ||r(i)||/||b|| 3.381476477625e-05
> > > > > > > > 246 KSP preconditioned resid norm 7.093960074437e-07 true resid norm 2.490667427871e-04 ||r(i)||/||b|| 3.381420626490e-05
> > > > > > > > 247 KSP preconditioned resid norm 7.817121711853e-07 true resid norm 2.490692299030e-04 ||r(i)||/||b|| 3.381454392480e-05
> > > > > > > > 248 KSP preconditioned resid norm 7.976109778309e-07 true resid norm 2.490686360729e-04 ||r(i)||/||b|| 3.381446330426e-05
> > > > > > > > 249 KSP preconditioned resid norm 7.855322750445e-07 true resid norm 2.490720861966e-04 ||r(i)||/||b|| 3.381493170560e-05
> > > > > > > > 250 KSP preconditioned resid norm 7.778531114042e-07 true resid norm 2.490673235034e-04 ||r(i)||/||b|| 3.381428510506e-05
> > > > > > > > 251 KSP preconditioned resid norm 7.848682182070e-07 true resid norm 2.490686360729e-04 ||r(i)||/||b|| 3.381446330426e-05
> > > > > > > > 252 KSP preconditioned resid norm 7.967291867330e-07 true resid norm 2.490724820229e-04 ||r(i)||/||b|| 3.381498544442e-05
> > > > > > > > 253 KSP preconditioned resid norm 7.865012959525e-07 true resid norm 2.490666662028e-04 ||r(i)||/||b|| 3.381419586754e-05
> > > > > > > > 254 KSP preconditioned resid norm 7.656025385804e-07 true resid norm 2.490686283214e-04 ||r(i)||/||b|| 3.381446225190e-05
> > > > > > > > 255 KSP preconditioned resid norm 7.757018653468e-07 true resid norm 2.490655983763e-04 ||r(i)||/||b|| 3.381405089553e-05
> > > > > > > > 256 KSP preconditioned resid norm 6.686490372981e-07 true resid norm 2.490715698964e-04 ||r(i)||/||b|| 3.381486161081e-05
> > > > > > > > 257 KSP preconditioned resid norm 6.596005109428e-07 true resid norm 2.490666403003e-04 ||r(i)||/||b|| 3.381419235092e-05
> > > > > > > > 258 KSP preconditioned resid norm 6.681742296333e-07 true resid norm 2.490683725835e-04 ||r(i)||/||b|| 3.381442753198e-05
> > > > > > > > 259 KSP preconditioned resid norm 1.089245482033e-06 true resid norm 2.490688086568e-04 ||r(i)||/||b|| 3.381448673488e-05
> > > > > > > > 260 KSP preconditioned resid norm 1.099844873189e-06 true resid norm 2.490690703265e-04 ||r(i)||/||b|| 3.381452226011e-05
> > > > > > > > 261 KSP preconditioned resid norm 1.112925540869e-06 true resid norm 2.490664481058e-04 ||r(i)||/||b|| 3.381416625790e-05
> > > > > > > > 262 KSP preconditioned resid norm 1.113056910480e-06 true resid norm 2.490658753273e-04 ||r(i)||/||b|| 3.381408849541e-05
> > > > > > > > 263 KSP preconditioned resid norm 1.104801535149e-06 true resid norm 2.490736510776e-04 ||r(i)||/||b|| 3.381514415953e-05
> > > > > > > > 264 KSP preconditioned resid norm 1.158709147873e-06 true resid norm 2.490607531152e-04 ||r(i)||/||b|| 3.381339308528e-05
> > > > > > > > 265 KSP preconditioned resid norm 1.178985740182e-06 true resid norm 2.490727895619e-04 ||r(i)||/||b|| 3.381502719703e-05
> > > > > > > > 266 KSP preconditioned resid norm 1.165130533478e-06 true resid norm 2.490639076693e-04 ||r(i)||/||b|| 3.381382135901e-05
> > > > > > > > 267 KSP preconditioned resid norm 1.181364114499e-06 true resid norm 2.490667871436e-04 ||r(i)||/||b|| 3.381421228690e-05
> > > > > > > > 268 KSP preconditioned resid norm 1.170295348543e-06 true resid norm 2.490662613306e-04 ||r(i)||/||b|| 3.381414090063e-05
> > > > > > > > 269 KSP preconditioned resid norm 1.213243016230e-06 true resid norm 2.490666173719e-04 ||r(i)||/||b|| 3.381418923808e-05
> > > > > > > > 270 KSP preconditioned resid norm 1.239691953997e-06 true resid norm 2.490678323197e-04 ||r(i)||/||b|| 3.381435418381e-05
> > > > > > > > 271 KSP preconditioned resid norm 1.219891740100e-06 true resid norm 2.490625009256e-04 ||r(i)||/||b|| 3.381363037437e-05
> > > > > > > > 272 KSP preconditioned resid norm 1.231321334346e-06 true resid norm 2.490659733696e-04 ||r(i)||/||b|| 3.381410180599e-05
> > > > > > > > 273 KSP preconditioned resid norm 1.208183234158e-06 true resid norm 2.490685987255e-04 ||r(i)||/||b|| 3.381445823385e-05
> > > > > > > > 274 KSP preconditioned resid norm 1.211768545589e-06 true resid norm 2.490671548953e-04 ||r(i)||/||b|| 3.381426221421e-05
> > > > > > > > 275 KSP preconditioned resid norm 1.209433459842e-06 true resid norm 2.490669016096e-04 ||r(i)||/||b|| 3.381422782722e-05
> > > > > > > > 276 KSP preconditioned resid norm 1.223729184405e-06 true resid norm 2.490658128014e-04 ||r(i)||/||b|| 3.381408000666e-05
> > > > > > > > 277 KSP preconditioned resid norm 1.243915201868e-06 true resid norm 2.490693375756e-04 ||r(i)||/||b|| 3.381455854282e-05
> > > > > > > > 278 KSP preconditioned resid norm 1.231994655529e-06 true resid norm 2.490682988311e-04 ||r(i)||/||b|| 3.381441751910e-05
> > > > > > > > 279 KSP preconditioned resid norm 1.227930683777e-06 true resid norm 2.490667825866e-04 ||r(i)||/||b|| 3.381421166823e-05
> > > > > > > > 280 KSP preconditioned resid norm 1.193458846469e-06 true resid norm 2.490687366117e-04 ||r(i)||/||b|| 3.381447695378e-05
> > > > > > > > 281 KSP preconditioned resid norm 1.217089059805e-06 true resid norm 2.490674797371e-04 ||r(i)||/||b|| 3.381430631591e-05
> > > > > > > > 282 KSP preconditioned resid norm 1.249318287709e-06 true resid norm 2.490662866951e-04 ||r(i)||/||b|| 3.381414434420e-05
> > > > > > > > 283 KSP preconditioned resid norm 1.183320029547e-06 true resid norm 2.490645783630e-04 ||r(i)||/||b|| 3.381391241482e-05
> > > > > > > > 284 KSP preconditioned resid norm 1.174730603102e-06 true resid norm 2.490686881647e-04 ||r(i)||/||b|| 3.381447037643e-05
> > > > > > > > 285 KSP preconditioned resid norm 1.175838261923e-06 true resid norm 2.490665969300e-04 ||r(i)||/||b|| 3.381418646281e-05
> > > > > > > > 286 KSP preconditioned resid norm 1.188946188368e-06 true resid norm 2.490661974622e-04 ||r(i)||/||b|| 3.381413222961e-05
> > > > > > > > 287 KSP preconditioned resid norm 1.177848565707e-06 true resid norm 2.490660236206e-04 ||r(i)||/||b|| 3.381410862824e-05
> > > > > > > > 288 KSP preconditioned resid norm 1.200075508281e-06 true resid norm 2.490645353536e-04 ||r(i)||/||b|| 3.381390657571e-05
> > > > > > > > 289 KSP preconditioned resid norm 1.184589570618e-06 true resid norm 2.490664920355e-04 ||r(i)||/||b|| 3.381417222195e-05
> > > > > > > > 290 KSP preconditioned resid norm 1.221114703873e-06 true resid norm 2.490670597538e-04 ||r(i)||/||b|| 3.381424929746e-05
> > > > > > > > 291 KSP preconditioned resid norm 1.249479658256e-06 true resid norm 2.490641582876e-04 ||r(i)||/||b|| 3.381385538385e-05
> > > > > > > > 292 KSP preconditioned resid norm 1.245768496850e-06 true resid norm 2.490704480588e-04 ||r(i)||/||b|| 3.381470930606e-05
> > > > > > > > 293 KSP preconditioned resid norm 1.243742607953e-06 true resid norm 2.490649690604e-04 ||r(i)||/||b|| 3.381396545733e-05
> > > > > > > > 294 KSP preconditioned resid norm 1.342758483339e-06 true resid norm 2.490676207432e-04 ||r(i)||/||b|| 3.381432545942e-05
> > > > > > > > 295 KSP preconditioned resid norm 1.353816099600e-06 true resid norm 2.490695263153e-04 ||r(i)||/||b|| 3.381458416681e-05
> > > > > > > > 296 KSP preconditioned resid norm 1.343886763293e-06 true resid norm 2.490673674307e-04 ||r(i)||/||b|| 3.381429106879e-05
> > > > > > > > 297 KSP preconditioned resid norm 1.355511022815e-06 true resid norm 2.490686565995e-04 ||r(i)||/||b|| 3.381446609103e-05
> > > > > > > > 298 KSP preconditioned resid norm 1.347247627243e-06 true resid norm 2.490696287707e-04 ||r(i)||/||b|| 3.381459807652e-05
> > > > > > > > 299 KSP preconditioned resid norm 1.414742595618e-06 true resid norm 2.490749815091e-04 ||r(i)||/||b|| 3.381532478374e-05
> > > > > > > > 300 KSP preconditioned resid norm 1.418560683189e-06 true resid norm 2.490721501153e-04 ||r(i)||/||b|| 3.381494038343e-05
> > > > > > > > 301 KSP preconditioned resid norm 1.416276404923e-06 true resid norm 2.490689576447e-04 ||r(i)||/||b|| 3.381450696203e-05
> > > > > > > > 302 KSP preconditioned resid norm 1.431448272112e-06 true resid norm 2.490688812701e-04 ||r(i)||/||b|| 3.381449659312e-05
> > > > > > > > 303 KSP preconditioned resid norm 1.446154958969e-06 true resid norm 2.490727536322e-04 ||r(i)||/||b|| 3.381502231909e-05
> > > > > > > > 304 KSP preconditioned resid norm 1.468860617921e-06 true resid norm 2.490692363788e-04 ||r(i)||/||b|| 3.381454480397e-05
> > > > > > > > 305 KSP preconditioned resid norm 1.627595214971e-06 true resid norm 2.490687019603e-04 ||r(i)||/||b|| 3.381447224938e-05
> > > > > > > > 306 KSP preconditioned resid norm 1.614384672893e-06 true resid norm 2.490687019603e-04 ||r(i)||/||b|| 3.381447224938e-05
> > > > > > > > 307 KSP preconditioned resid norm 1.605568020532e-06 true resid norm 2.490699757693e-04 ||r(i)||/||b|| 3.381464518632e-05
> > > > > > > > 308 KSP preconditioned resid norm 1.617069685075e-06 true resid norm 2.490649282923e-04 ||r(i)||/||b|| 3.381395992249e-05
> > > > > > > > 309 KSP preconditioned resid norm 1.654297792738e-06 true resid norm 2.490644766626e-04 ||r(i)||/||b|| 3.381389860760e-05
> > > > > > > > 310 KSP preconditioned resid norm 1.587528143215e-06 true resid norm 2.490696752096e-04 ||r(i)||/||b|| 3.381460438124e-05
> > > > > > > > 311 KSP preconditioned resid norm 1.662782022388e-06 true resid norm 2.490699317737e-04 ||r(i)||/||b|| 3.381463921332e-05
> > > > > > > > 312 KSP preconditioned resid norm 1.618211471748e-06 true resid norm 2.490735831308e-04 ||r(i)||/||b|| 3.381513493483e-05
> > > > > > > > 313 KSP preconditioned resid norm 1.609074961921e-06 true resid norm 2.490679566436e-04 ||r(i)||/||b|| 3.381437106247e-05
> > > > > > > > 314 KSP preconditioned resid norm 1.548068942878e-06 true resid norm 2.490660071226e-04 ||r(i)||/||b|| 3.381410638842e-05
> > > > > > > > 315 KSP preconditioned resid norm 1.526718322150e-06 true resid norm 2.490619832967e-04 ||r(i)||/||b|| 3.381356009919e-05
> > > > > > > > 316 KSP preconditioned resid norm 1.553150959105e-06 true resid norm 2.490660071226e-04 ||r(i)||/||b|| 3.381410638842e-05
> > > > > > > > 317 KSP preconditioned resid norm 1.615015320906e-06 true resid norm 2.490672348079e-04 ||r(i)||/||b|| 3.381427306343e-05
> > > > > > > > 318 KSP preconditioned resid norm 1.602904469797e-06 true resid norm 2.490696731006e-04 ||r(i)||/||b|| 3.381460409491e-05
> > > > > > > > 319 KSP preconditioned resid norm 1.538140323073e-06 true resid norm 2.490722982494e-04 ||r(i)||/||b|| 3.381496049466e-05
> > > > > > > > 320 KSP preconditioned resid norm 1.534779679430e-06 true resid norm 2.490778499789e-04 ||r(i)||/||b|| 3.381571421763e-05
> > > > > > > > 321 KSP preconditioned resid norm 1.547155843355e-06 true resid norm 2.490767612985e-04 ||r(i)||/||b|| 3.381556641442e-05
> > > > > > > > 322 KSP preconditioned resid norm 1.422137008870e-06 true resid norm 2.490737676309e-04 ||r(i)||/||b|| 3.381515998323e-05
> > > > > > > > 323 KSP preconditioned resid norm 1.403072558954e-06 true resid norm 2.490741361870e-04 ||r(i)||/||b|| 3.381521001975e-05
> > > > > > > > 324 KSP preconditioned resid norm 1.373070436118e-06 true resid norm 2.490742214990e-04 ||r(i)||/||b|| 3.381522160202e-05
> > > > > > > > 325 KSP preconditioned resid norm 1.359547585233e-06 true resid norm 2.490792987570e-04 ||r(i)||/||b|| 3.381591090902e-05
> > > > > > > > 326 KSP preconditioned resid norm 1.370351913612e-06 true resid norm 2.490727161158e-04 ||r(i)||/||b|| 3.381501722573e-05
> > > > > > > > 327 KSP preconditioned resid norm 1.365238666187e-06 true resid norm 2.490716949642e-04 ||r(i)||/||b|| 3.381487859046e-05
> > > > > > > > 328 KSP preconditioned resid norm 1.369073373042e-06 true resid norm 2.490807360288e-04 ||r(i)||/||b|| 3.381610603826e-05
> > > > > > > > 329 KSP preconditioned resid norm 1.426698981572e-06 true resid norm 2.490791479521e-04 ||r(i)||/||b|| 3.381589043520e-05
> > > > > > > > 330 KSP preconditioned resid norm 1.445542403570e-06 true resid norm 2.490775981409e-04 ||r(i)||/||b|| 3.381568002720e-05
> > > > > > > > 331 KSP preconditioned resid norm 1.464506963984e-06 true resid norm 2.490740562430e-04 ||r(i)||/||b|| 3.381519916626e-05
> > > > > > > > 332 KSP preconditioned resid norm 1.461462964401e-06 true resid norm 2.490768016856e-04 ||r(i)||/||b|| 3.381557189753e-05
> > > > > > > > 333 KSP preconditioned resid norm 1.476680847971e-06 true resid norm 2.490744321516e-04 ||r(i)||/||b|| 3.381525020097e-05
> > > > > > > > 334 KSP preconditioned resid norm 1.459640372198e-06 true resid norm 2.490788817993e-04 ||r(i)||/||b|| 3.381585430132e-05
> > > > > > > > 335 KSP preconditioned resid norm 1.790770882365e-06 true resid norm 2.490771711471e-04 ||r(i)||/||b|| 3.381562205697e-05
> > > > > > > > 336 KSP preconditioned resid norm 1.803770155018e-06 true resid norm 2.490768953858e-04 ||r(i)||/||b|| 3.381558461860e-05
> > > > > > > > 337 KSP preconditioned resid norm 1.787821255995e-06 true resid norm 2.490767985676e-04 ||r(i)||/||b|| 3.381557147421e-05
> > > > > > > > 338 KSP preconditioned resid norm 1.749912220831e-06 true resid norm 2.490760198704e-04 ||r(i)||/||b|| 3.381546575545e-05
> > > > > > > > 339 KSP preconditioned resid norm 1.802915839010e-06 true resid norm 2.490815556273e-04 ||r(i)||/||b|| 3.381621730993e-05
> > > > > > > > 340 KSP preconditioned resid norm 1.800777670709e-06 true resid norm 2.490823909286e-04 ||r(i)||/||b|| 3.381633071347e-05
> > > > > > > > 341 KSP preconditioned resid norm 1.962516327690e-06 true resid norm 2.490773477410e-04 ||r(i)||/||b|| 3.381564603199e-05
> > > > > > > > 342 KSP preconditioned resid norm 1.981726465132e-06 true resid norm 2.490769116884e-04 ||r(i)||/||b|| 3.381558683191e-05
> > > > > > > > 343 KSP preconditioned resid norm 1.963419167052e-06 true resid norm 2.490764009914e-04 ||r(i)||/||b|| 3.381551749783e-05
> > > > > > > > 344 KSP preconditioned resid norm 1.992082169278e-06 true resid norm 2.490806829883e-04 ||r(i)||/||b|| 3.381609883728e-05
> > > > > > > > 345 KSP preconditioned resid norm 1.981005134253e-06 true resid norm 2.490748677339e-04 ||r(i)||/||b|| 3.381530933721e-05
> > > > > > > > 346 KSP preconditioned resid norm 1.959802663114e-06 true resid norm 2.490773752317e-04 ||r(i)||/||b|| 3.381564976423e-05
> > > > > > > >
> > > > > > > > On Sat, Oct 21, 2017 at 5:25 PM, Matthew Knepley <knepley at gmail.com> wrote:
> > > > > > > > On Sat, Oct 21, 2017 at 5:21 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > > > > > > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
> > > > > > > > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
> > > > > > > >
> > > > > > > > ierr = VecAssemblyBegin(x);
> > > > > > > > ierr = VecAssemblyEnd(x);
> > > > > > > > This is probably unnecessary
> > > > > > > >
> > > > > > > > ierr = VecAssemblyBegin(b);
> > > > > > > > ierr = VecAssemblyEnd(b);
> > > > > > > > This is probably unnecessary
> > > > > > > >
> > > > > > > >
> > > > > > > > ierr = MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL,&nullsp);
> > > > > > > > ierr = MatSetNullSpace(A,nullsp); // Petsc-3.8
> > > > > > > > Is your rhs consistent with this nullspace?
> > > > > > > >
> > > > > > > > // KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);
> > > > > > > > KSPSetOperators(ksp,A,A);
> > > > > > > >
> > > > > > > > KSPSetType(ksp,KSPBCGS);
> > > > > > > >
> > > > > > > > KSPSetComputeSingularValues(ksp, PETSC_TRUE);
> > > > > > > > #if defined(__HYPRE__)
> > > > > > > > KSPGetPC(ksp, &pc);
> > > > > > > > PCSetType(pc, PCHYPRE);
> > > > > > > > PCHYPRESetType(pc,"boomeramg");
> > > > > > > > This is terribly unnecessary. You just use
> > > > > > > >
> > > > > > > > -pc_type hypre -pc_hypre_type boomeramg
> > > > > > > >
> > > > > > > > or
> > > > > > > >
> > > > > > > > -pc_type gamg
> > > > > > > >
> > > > > > > > #else
> > > > > > > > KSPSetType(ksp,KSPBCGSL);
> > > > > > > > KSPBCGSLSetEll(ksp,2);
> > > > > > > > #endif /* defined(__HYPRE__) */
> > > > > > > >
> > > > > > > > KSPSetFromOptions(ksp);
> > > > > > > > KSPSetUp(ksp);
> > > > > > > >
> > > > > > > > ierr = KSPSolve(ksp,b,x);
> > > > > > > >
> > > > > > > >
> > > > > > > > command line
> > > > > > > >
> > > > > > > > You did not provide any of what I asked for the in the eprevious mail.
> > > > > > > >
> > > > > > > > Matt
> > > > > > > >
> > > > > > > > On Sat, Oct 21, 2017 at 5:16 PM, Matthew Knepley <knepley at gmail.com> wrote:
> > > > > > > > On Sat, Oct 21, 2017 at 5:04 PM, Hao Zhang <hbcbh1999 at gmail.com> wrote:
> > > > > > > > hi,
> > > > > > > >
> > > > > > > > I implemented HYPRE preconditioner for my study due to the fact that without preconditioner, PETSc solver will take thousands of iterations to converge for fine grid simulation.
> > > > > > > >
> > > > > > > > with HYPRE, depending on the parallel partition, it will take HYPRE forever to do anything. observation of output file is that the simulation is hanging with no output.
> > > > > > > >
> > > > > > > > Any idea what happened? will post snippet of code.
> > > > > > > >
> > > > > > > > 1) For any question about convergence, we need to see the output of
> > > > > > > >
> > > > > > > > -ksp_view_pre -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
> > > > > > > >
> > > > > > > > 2) Hypre has many preconditioners, which one are you talking about
> > > > > > > >
> > > > > > > > 3) PETSc has some preconditioners in common with Hypre, like AMG
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Matt
> > > > > > > >
> > > > > > > > --
> > > > > > > > Hao Zhang
> > > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > > Stony Brook University,
> > > > > > > > Stony Brook, New York, 11790
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > > > > > > > -- Norbert Wiener
> > > > > > > >
> > > > > > > > https://www.cse.buffalo.edu/~knepley/
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Hao Zhang
> > > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > > Stony Brook University,
> > > > > > > > Stony Brook, New York, 11790
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > > > > > > > -- Norbert Wiener
> > > > > > > >
> > > > > > > > https://www.cse.buffalo.edu/~knepley/
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Hao Zhang
> > > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > > Stony Brook University,
> > > > > > > > Stony Brook, New York, 11790
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Hao Zhang
> > > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > > Stony Brook University,
> > > > > > > > Stony Brook, New York, 11790
> > > > > > >
> > > > > > > --
> > > > > > > Hao Zhang
> > > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > > Stony Brook University,
> > > > > > > Stony Brook, New York, 11790
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Hao Zhang
> > > > > > Dept. of Applid Mathematics and Statistics,
> > > > > > Stony Brook University,
> > > > > > Stony Brook, New York, 11790
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Hao Zhang
> > > > > Dept. of Applid Mathematics and Statistics,
> > > > > Stony Brook University,
> > > > > Stony Brook, New York, 11790
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Hao Zhang
> > > > Dept. of Applid Mathematics and Statistics,
> > > > Stony Brook University,
> > > > Stony Brook, New York, 11790
> > >
> > >
> > >
> > >
> > > --
> > > Hao Zhang
> > > Dept. of Applid Mathematics and Statistics,
> > > Stony Brook University,
> > > Stony Brook, New York, 11790
> >
> >
> >
> >
> > --
> > Hao Zhang
> > Dept. of Applid Mathematics and Statistics,
> > Stony Brook University,
> > Stony Brook, New York, 11790
>
>
>
>
>
> --
> Hao Zhang
> Dept. of Applid Mathematics and Statistics,
> Stony Brook University,
> Stony Brook, New York, 11790
>
>
>
>
> --
> Hao Zhang
> Dept. of Applid Mathematics and Statistics,
> Stony Brook University,
> Stony Brook, New York, 11790
More information about the petsc-users
mailing list