From filippo.leonardi at sam.math.ethz.ch  Tue Sep  1 02:22:11 2015
From: filippo.leonardi at sam.math.ethz.ch (Filippo Leonardi)
Date: Tue, 1 Sep 2015 09:22:11 +0200
Subject: [petsc-users] 2D Poisson on nonuniform meshes
Message-ID: <3634751.Hfn5OcL7V3@besikowitch-iii>

Dear PETSc Users,

I want to use multigrid to solve uniform (just Laplace) poisson in 2D/3D on cartesian, non-
uniform meshes with a standard 5 (7)-points stencil FD.

I always scaled my Poisson matrix like in the doc examples, i.e. multiplying by dx*dy (so that in 
ComputeRHS I need to scale b, in A*x = b, as well). This always worked properly with both MG/
GAMG and with galerkin matrices.

Now I'd like to use non-uniform meshes, therefore the scaling is non-uniform. However I cannot 
get my matrices to scale properly with any sort of multigrid.

One would think that without scaling, i.e. solving the original system, at least MG+galerkin or 
GAMG should work anyways provided the matrix A and b are consistent.

I tried without boundaries (i.e. torus), so this is not the problem.

Anyone did/knows how to do this properly?

Thanks,
F
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/89a3c005/attachment.html>

From jychang48 at gmail.com  Tue Sep  1 03:11:04 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 1 Sep 2015 02:11:04 -0600
Subject: [petsc-users] Integrating TAO into SNES and TS
In-Reply-To: <C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
References: <CAP2=TMgF8fKyX0o--rRz+cw01=d5ws6iCZu=2PVq1qBJq50okA@mail.gmail.com>
	<16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov>
	<CAP2=TMhD41MWMf=n+Z7ZdDdNYvRVx91fL956AXSjtJb_vz01gg@mail.gmail.com>
	<0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov>
	<CAP2=TMh=NSnObLKwgR_ZJHZDqNe_c+VO4NPkSFnA3X+NLd0Amg@mail.gmail.com>
	<6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov>
	<CAP2=TMiZXce6FBKMRNBtnyn3Ug7cnMn_2ZhgR2547=OqpO8Nog@mail.gmail.com>
	<C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
Message-ID: <CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>

Barry,

That's good to know thanks.

On a related note, is it possible for VI to one day include linear equality
constraints?

Thanks,
Justin

On Mon, Aug 31, 2015 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Aug 31, 2015, at 7:36 PM, Justin Chang <jychang48 at gmail.com> wrote:
> >
> > Coming back to this,
> >
> > Say I now want to ensure the DMP for advection-diffusion equations. The
> linear operator is now asymmetric and non-self-adjoint (assuming I do
> something like SUPG or finite volume), meaning I cannot simply solve this
> problem without any manipulation (e.g. normalizing the equations) using
> TAO's optimization solvers. Does this statement also hold true for SNESVI?
>
>   SNESVI doesn't care about symmetry etc
>
> >
> > Thanks,
> > Justin
> >
> > On Fri, Apr 3, 2015 at 7:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Apr 3, 2015, at 7:35 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > >
> > > I guess I will have to write my own code then :)
> > >
> > > I am not all that familiar with Variational Inequalities at the
> moment, but if my Jacobian is symmetric and positive definite and I only
> have lower and upper bounds, doesn't the problem simply reduce to that of a
> convex optimization? That is, with SNES act as if it were Tao?
> >
> >   Yes, I think that is essentially correctly.
> >
> >   Barry
> >
> > >
> > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >   Justin,
> > >
> > >    We haven't done anything with TS to handle variational
> inequalities. So you can either write your own backward Euler (outside of
> TS) that solves each time-step problem either as 1) an optimization problem
> using Tao or 2) as a variational inequality using SNES.
> > >
> > >    More adventurously you could look at the TSTHETA code in TS (which
> is a general form that includes Euler, Backward Euler and Crank-Nicolson
> and see if you can add the constraints to the SNES problem that is solved;
> in theory this is straightforward but it would require understanding the
> current code (which Jed, of course, overwrote :-). I think you should do
> this.
> > >
> > >   Barry
> > >
> > >
> > > > On Apr 3, 2015, at 12:31 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > >
> > > > I am solving the following anisotropic transient diffusion equation
> subject to 0 bounds:
> > > >
> > > > du/dt = div[D*grad[u]] + f
> > > >
> > > > Where the dispersion tensor D(x) is symmetric and positive definite.
> This formulation violates the discrete maximum principles so one of the
> ways to ensure nonnegative concentrations is to employ convex optimization.
> I am following the procedures in Nakshatrala and Valocchi (2009) JCP and
> Nagarajan and Nakshatrala (2011) IJNMF.
> > > >
> > > > The Variational Inequality method works gives what I want for my
> transient case, but what if I want to implement the Tao methodology in TS?
> That is, what TS functions do I need to set up steps a) through e) for each
> time step (also the Jacobian remains the same for all time steps so I would
> only call this once). Normally I would just call TSSolve() and let the
> libraries and functions do everything, but I would like to incorporate
> TaoSolve into every time step.
> > > >
> > > > Thanks,
> > > >
> > > > --
> > > > Justin Chang
> > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > Houston, TX 77004
> > > > (512) 963-3262
> > > >
> > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > >
> > > >   An alternative approach is for you to solve it as a (non)linear
> variational inequality. See src/snes/examples/tutorials/ex9.c
> > > >
> > > >   How you should proceed depends on your long term goal. What
> problem do you really want to solve? Is it really a linear time dependent
> problem with 0 bounds on U? Can the problem always be represented as an
> optimization problem easily? What are  and what will be the properties of
> K? For example if K is positive definite then likely the bounds will remain
> try without explicitly providing the constraints.
> > > >
> > > >   Barry
> > > >
> > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I have a two part question regarding the integration of the
> following optimization problem
> > > > >
> > > > > min 1/2 u^T*K*u + u^T*f
> > > > > S.T. u >= 0
> > > > >
> > > > > into SNES and TS
> > > > >
> > > > > 1) For SNES, assuming I am working with a linear FE equation, I
> have the following algorithm/steps for solving my problem
> > > > >
> > > > > a) Set an initial guess x
> > > > > b) Obtain residual r and jacobian A through functions
> SNESComputeFunction() and SNESComputeJacobian() respectively
> > > > > c) Form vector b = r - A*x
> > > > > d) Set Hessian equal to A, gradient to A*x, objective function
> value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector
> > > > > e) Call TaoSolve
> > > > >
> > > > > This works well at the moment, but my question is there a more
> "efficient" way of doing this? Because with my current setup, I am making a
> rather bold assumption that my problem would converge in one SNES iteration
> without the bounded constraints and does not have any unexpected
> nonlinearities.
> > > > >
> > > > > 2) How would I go about doing the above for time-stepping
> problems? At each time step, I want to solve a convex optimization subject
> to the lower bounds constraint. I plan on using backward euler and my
> resulting jacobian should still be compatible with the above optimization
> problem.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > --
> > > > > Justin Chang
> > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > > Houston, TX 77004
> > > > > (512) 963-3262
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Justin Chang
> > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > Houston, TX 77004
> > > > (512) 963-3262
> > >
> > >
> > >
> > >
> > > --
> > > Justin Chang
> > > PhD Candidate, Civil Engineering - Computational Sciences
> > > University of Houston, Department of Civil and Environmental
> Engineering
> > > Houston, TX 77004
> > > (512) 963-3262
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/d1b98bb1/attachment.html>

From knepley at gmail.com  Tue Sep  1 03:33:19 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 Sep 2015 03:33:19 -0500
Subject: [petsc-users] Integrating TAO into SNES and TS
In-Reply-To: <CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>
References: <CAP2=TMgF8fKyX0o--rRz+cw01=d5ws6iCZu=2PVq1qBJq50okA@mail.gmail.com>
	<16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov>
	<CAP2=TMhD41MWMf=n+Z7ZdDdNYvRVx91fL956AXSjtJb_vz01gg@mail.gmail.com>
	<0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov>
	<CAP2=TMh=NSnObLKwgR_ZJHZDqNe_c+VO4NPkSFnA3X+NLd0Amg@mail.gmail.com>
	<6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov>
	<CAP2=TMiZXce6FBKMRNBtnyn3Ug7cnMn_2ZhgR2547=OqpO8Nog@mail.gmail.com>
	<C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
	<CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>
Message-ID: <CAMYG4G=MTDKpRrfnV+nBZvCGWNhTe-qtSkmVoCOF4J_uLS=e4A@mail.gmail.com>

On Tue, Sep 1, 2015 at 3:11 AM, Justin Chang <jychang48 at gmail.com> wrote:

> Barry,
>
> That's good to know thanks.
>
> On a related note, is it possible for VI to one day include linear
> equality constraints?
>

How are these different from just using more equations?

  Thanks,

    Matt


> Thanks,
> Justin
>
> On Mon, Aug 31, 2015 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> > On Aug 31, 2015, at 7:36 PM, Justin Chang <jychang48 at gmail.com> wrote:
>> >
>> > Coming back to this,
>> >
>> > Say I now want to ensure the DMP for advection-diffusion equations. The
>> linear operator is now asymmetric and non-self-adjoint (assuming I do
>> something like SUPG or finite volume), meaning I cannot simply solve this
>> problem without any manipulation (e.g. normalizing the equations) using
>> TAO's optimization solvers. Does this statement also hold true for SNESVI?
>>
>>   SNESVI doesn't care about symmetry etc
>>
>> >
>> > Thanks,
>> > Justin
>> >
>> > On Fri, Apr 3, 2015 at 7:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> >
>> > > On Apr 3, 2015, at 7:35 PM, Justin Chang <jchang27 at uh.edu> wrote:
>> > >
>> > > I guess I will have to write my own code then :)
>> > >
>> > > I am not all that familiar with Variational Inequalities at the
>> moment, but if my Jacobian is symmetric and positive definite and I only
>> have lower and upper bounds, doesn't the problem simply reduce to that of a
>> convex optimization? That is, with SNES act as if it were Tao?
>> >
>> >   Yes, I think that is essentially correctly.
>> >
>> >   Barry
>> >
>> > >
>> > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> > >
>> > >   Justin,
>> > >
>> > >    We haven't done anything with TS to handle variational
>> inequalities. So you can either write your own backward Euler (outside of
>> TS) that solves each time-step problem either as 1) an optimization problem
>> using Tao or 2) as a variational inequality using SNES.
>> > >
>> > >    More adventurously you could look at the TSTHETA code in TS (which
>> is a general form that includes Euler, Backward Euler and Crank-Nicolson
>> and see if you can add the constraints to the SNES problem that is solved;
>> in theory this is straightforward but it would require understanding the
>> current code (which Jed, of course, overwrote :-). I think you should do
>> this.
>> > >
>> > >   Barry
>> > >
>> > >
>> > > > On Apr 3, 2015, at 12:31 PM, Justin Chang <jchang27 at uh.edu> wrote:
>> > > >
>> > > > I am solving the following anisotropic transient diffusion equation
>> subject to 0 bounds:
>> > > >
>> > > > du/dt = div[D*grad[u]] + f
>> > > >
>> > > > Where the dispersion tensor D(x) is symmetric and positive
>> definite. This formulation violates the discrete maximum principles so one
>> of the ways to ensure nonnegative concentrations is to employ convex
>> optimization. I am following the procedures in Nakshatrala and Valocchi
>> (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF.
>> > > >
>> > > > The Variational Inequality method works gives what I want for my
>> transient case, but what if I want to implement the Tao methodology in TS?
>> That is, what TS functions do I need to set up steps a) through e) for each
>> time step (also the Jacobian remains the same for all time steps so I would
>> only call this once). Normally I would just call TSSolve() and let the
>> libraries and functions do everything, but I would like to incorporate
>> TaoSolve into every time step.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > --
>> > > > Justin Chang
>> > > > PhD Candidate, Civil Engineering - Computational Sciences
>> > > > University of Houston, Department of Civil and Environmental
>> Engineering
>> > > > Houston, TX 77004
>> > > > (512) 963-3262
>> > > >
>> > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> > > >
>> > > >   An alternative approach is for you to solve it as a (non)linear
>> variational inequality. See src/snes/examples/tutorials/ex9.c
>> > > >
>> > > >   How you should proceed depends on your long term goal. What
>> problem do you really want to solve? Is it really a linear time dependent
>> problem with 0 bounds on U? Can the problem always be represented as an
>> optimization problem easily? What are  and what will be the properties of
>> K? For example if K is positive definite then likely the bounds will remain
>> try without explicitly providing the constraints.
>> > > >
>> > > >   Barry
>> > > >
>> > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang <jchang27 at uh.edu> wrote:
>> > > > >
>> > > > > Hi everyone,
>> > > > >
>> > > > > I have a two part question regarding the integration of the
>> following optimization problem
>> > > > >
>> > > > > min 1/2 u^T*K*u + u^T*f
>> > > > > S.T. u >= 0
>> > > > >
>> > > > > into SNES and TS
>> > > > >
>> > > > > 1) For SNES, assuming I am working with a linear FE equation, I
>> have the following algorithm/steps for solving my problem
>> > > > >
>> > > > > a) Set an initial guess x
>> > > > > b) Obtain residual r and jacobian A through functions
>> SNESComputeFunction() and SNESComputeJacobian() respectively
>> > > > > c) Form vector b = r - A*x
>> > > > > d) Set Hessian equal to A, gradient to A*x, objective function
>> value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector
>> > > > > e) Call TaoSolve
>> > > > >
>> > > > > This works well at the moment, but my question is there a more
>> "efficient" way of doing this? Because with my current setup, I am making a
>> rather bold assumption that my problem would converge in one SNES iteration
>> without the bounded constraints and does not have any unexpected
>> nonlinearities.
>> > > > >
>> > > > > 2) How would I go about doing the above for time-stepping
>> problems? At each time step, I want to solve a convex optimization subject
>> to the lower bounds constraint. I plan on using backward euler and my
>> resulting jacobian should still be compatible with the above optimization
>> problem.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > --
>> > > > > Justin Chang
>> > > > > PhD Candidate, Civil Engineering - Computational Sciences
>> > > > > University of Houston, Department of Civil and Environmental
>> Engineering
>> > > > > Houston, TX 77004
>> > > > > (512) 963-3262
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Justin Chang
>> > > > PhD Candidate, Civil Engineering - Computational Sciences
>> > > > University of Houston, Department of Civil and Environmental
>> Engineering
>> > > > Houston, TX 77004
>> > > > (512) 963-3262
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > > Justin Chang
>> > > PhD Candidate, Civil Engineering - Computational Sciences
>> > > University of Houston, Department of Civil and Environmental
>> Engineering
>> > > Houston, TX 77004
>> > > (512) 963-3262
>> >
>> >
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/7697963e/attachment-0001.html>

From knepley at gmail.com  Tue Sep  1 03:40:14 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 Sep 2015 03:40:14 -0500
Subject: [petsc-users] 2D Poisson on nonuniform meshes
In-Reply-To: <3634751.Hfn5OcL7V3@besikowitch-iii>
References: <3634751.Hfn5OcL7V3@besikowitch-iii>
Message-ID: <CAMYG4Gm1KATfJAEJeZT_AGpwyS+Vd5RYLfXpda=JMANtd_ANrQ@mail.gmail.com>

On Tue, Sep 1, 2015 at 2:22 AM, Filippo Leonardi <
filippo.leonardi at sam.math.ethz.ch> wrote:

> Dear PETSc Users,
>
>
>
> I want to use multigrid to solve uniform (just Laplace) poisson in 2D/3D
> on cartesian, non-uniform meshes with a standard 5 (7)-points stencil FD.
>
>
>
> I always scaled my Poisson matrix like in the doc examples, i.e.
> multiplying by dx*dy (so that in ComputeRHS I need to scale b, in A*x = b,
> as well). This always worked properly with both MG/GAMG and with galerkin
> matrices.
>
>
>
> Now I'd like to use non-uniform meshes, therefore the scaling is
> non-uniform. However I cannot get my matrices to scale properly with any
> sort of multigrid.
>
>
>
> One would think that without scaling, i.e. solving the original system, at
> least MG+galerkin or GAMG should work anyways provided the matrix A and b
> are consistent.
>
>
>
> I tried without boundaries (i.e. torus), so this is not the problem.
>

The point of that scaling is to make the boundary values the same size as
the residuals. If you have
no boundary values, the problem is scale invariant since its linear. It
would be nice to scale it so everything
is about size 1.

When asking about convergence always run with -ksp_view
-ksp_monitor_true_residual -ksp_converged_reason

  Thanks,

     Matt


> Anyone did/knows how to do this properly?
>
>
>
> Thanks,
>
> F
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/a005baa0/attachment.html>

From jychang48 at gmail.com  Tue Sep  1 03:46:32 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 1 Sep 2015 02:46:32 -0600
Subject: [petsc-users] Integrating TAO into SNES and TS
In-Reply-To: <CAMYG4G=MTDKpRrfnV+nBZvCGWNhTe-qtSkmVoCOF4J_uLS=e4A@mail.gmail.com>
References: <CAP2=TMgF8fKyX0o--rRz+cw01=d5ws6iCZu=2PVq1qBJq50okA@mail.gmail.com>
	<16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov>
	<CAP2=TMhD41MWMf=n+Z7ZdDdNYvRVx91fL956AXSjtJb_vz01gg@mail.gmail.com>
	<0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov>
	<CAP2=TMh=NSnObLKwgR_ZJHZDqNe_c+VO4NPkSFnA3X+NLd0Amg@mail.gmail.com>
	<6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov>
	<CAP2=TMiZXce6FBKMRNBtnyn3Ug7cnMn_2ZhgR2547=OqpO8Nog@mail.gmail.com>
	<C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
	<CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>
	<CAMYG4G=MTDKpRrfnV+nBZvCGWNhTe-qtSkmVoCOF4J_uLS=e4A@mail.gmail.com>
Message-ID: <CAP2=TMjdp3iKJ8943dHwKQmFg1cb9WOt29MgEgaiNTmMMcmoow@mail.gmail.com>

I would like to simultaneously enforce both discrete maximum principle and
local mass/species balance. Because even if a locally conservative scheme
like RT0 is used, as soon as these bounded constraints are applied, i lose
the mass balance.

On Tue, Sep 1, 2015 at 2:33 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Sep 1, 2015 at 3:11 AM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> Barry,
>>
>> That's good to know thanks.
>>
>> On a related note, is it possible for VI to one day include linear
>> equality constraints?
>>
>
> How are these different from just using more equations?
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>> Justin
>>
>> On Mon, Aug 31, 2015 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>> > On Aug 31, 2015, at 7:36 PM, Justin Chang <jychang48 at gmail.com> wrote:
>>> >
>>> > Coming back to this,
>>> >
>>> > Say I now want to ensure the DMP for advection-diffusion equations.
>>> The linear operator is now asymmetric and non-self-adjoint (assuming I do
>>> something like SUPG or finite volume), meaning I cannot simply solve this
>>> problem without any manipulation (e.g. normalizing the equations) using
>>> TAO's optimization solvers. Does this statement also hold true for SNESVI?
>>>
>>>   SNESVI doesn't care about symmetry etc
>>>
>>> >
>>> > Thanks,
>>> > Justin
>>> >
>>> > On Fri, Apr 3, 2015 at 7:38 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> >
>>> > > On Apr 3, 2015, at 7:35 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>> > >
>>> > > I guess I will have to write my own code then :)
>>> > >
>>> > > I am not all that familiar with Variational Inequalities at the
>>> moment, but if my Jacobian is symmetric and positive definite and I only
>>> have lower and upper bounds, doesn't the problem simply reduce to that of a
>>> convex optimization? That is, with SNES act as if it were Tao?
>>> >
>>> >   Yes, I think that is essentially correctly.
>>> >
>>> >   Barry
>>> >
>>> > >
>>> > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> > >
>>> > >   Justin,
>>> > >
>>> > >    We haven't done anything with TS to handle variational
>>> inequalities. So you can either write your own backward Euler (outside of
>>> TS) that solves each time-step problem either as 1) an optimization problem
>>> using Tao or 2) as a variational inequality using SNES.
>>> > >
>>> > >    More adventurously you could look at the TSTHETA code in TS
>>> (which is a general form that includes Euler, Backward Euler and
>>> Crank-Nicolson and see if you can add the constraints to the SNES problem
>>> that is solved; in theory this is straightforward but it would require
>>> understanding the current code (which Jed, of course, overwrote :-). I
>>> think you should do this.
>>> > >
>>> > >   Barry
>>> > >
>>> > >
>>> > > > On Apr 3, 2015, at 12:31 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>> > > >
>>> > > > I am solving the following anisotropic transient diffusion
>>> equation subject to 0 bounds:
>>> > > >
>>> > > > du/dt = div[D*grad[u]] + f
>>> > > >
>>> > > > Where the dispersion tensor D(x) is symmetric and positive
>>> definite. This formulation violates the discrete maximum principles so one
>>> of the ways to ensure nonnegative concentrations is to employ convex
>>> optimization. I am following the procedures in Nakshatrala and Valocchi
>>> (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF.
>>> > > >
>>> > > > The Variational Inequality method works gives what I want for my
>>> transient case, but what if I want to implement the Tao methodology in TS?
>>> That is, what TS functions do I need to set up steps a) through e) for each
>>> time step (also the Jacobian remains the same for all time steps so I would
>>> only call this once). Normally I would just call TSSolve() and let the
>>> libraries and functions do everything, but I would like to incorporate
>>> TaoSolve into every time step.
>>> > > >
>>> > > > Thanks,
>>> > > >
>>> > > > --
>>> > > > Justin Chang
>>> > > > PhD Candidate, Civil Engineering - Computational Sciences
>>> > > > University of Houston, Department of Civil and Environmental
>>> Engineering
>>> > > > Houston, TX 77004
>>> > > > (512) 963-3262
>>> > > >
>>> > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> > > >
>>> > > >   An alternative approach is for you to solve it as a (non)linear
>>> variational inequality. See src/snes/examples/tutorials/ex9.c
>>> > > >
>>> > > >   How you should proceed depends on your long term goal. What
>>> problem do you really want to solve? Is it really a linear time dependent
>>> problem with 0 bounds on U? Can the problem always be represented as an
>>> optimization problem easily? What are  and what will be the properties of
>>> K? For example if K is positive definite then likely the bounds will remain
>>> try without explicitly providing the constraints.
>>> > > >
>>> > > >   Barry
>>> > > >
>>> > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang <jchang27 at uh.edu>
>>> wrote:
>>> > > > >
>>> > > > > Hi everyone,
>>> > > > >
>>> > > > > I have a two part question regarding the integration of the
>>> following optimization problem
>>> > > > >
>>> > > > > min 1/2 u^T*K*u + u^T*f
>>> > > > > S.T. u >= 0
>>> > > > >
>>> > > > > into SNES and TS
>>> > > > >
>>> > > > > 1) For SNES, assuming I am working with a linear FE equation, I
>>> have the following algorithm/steps for solving my problem
>>> > > > >
>>> > > > > a) Set an initial guess x
>>> > > > > b) Obtain residual r and jacobian A through functions
>>> SNESComputeFunction() and SNESComputeJacobian() respectively
>>> > > > > c) Form vector b = r - A*x
>>> > > > > d) Set Hessian equal to A, gradient to A*x, objective function
>>> value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector
>>> > > > > e) Call TaoSolve
>>> > > > >
>>> > > > > This works well at the moment, but my question is there a more
>>> "efficient" way of doing this? Because with my current setup, I am making a
>>> rather bold assumption that my problem would converge in one SNES iteration
>>> without the bounded constraints and does not have any unexpected
>>> nonlinearities.
>>> > > > >
>>> > > > > 2) How would I go about doing the above for time-stepping
>>> problems? At each time step, I want to solve a convex optimization subject
>>> to the lower bounds constraint. I plan on using backward euler and my
>>> resulting jacobian should still be compatible with the above optimization
>>> problem.
>>> > > > >
>>> > > > > Thanks,
>>> > > > >
>>> > > > > --
>>> > > > > Justin Chang
>>> > > > > PhD Candidate, Civil Engineering - Computational Sciences
>>> > > > > University of Houston, Department of Civil and Environmental
>>> Engineering
>>> > > > > Houston, TX 77004
>>> > > > > (512) 963-3262
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Justin Chang
>>> > > > PhD Candidate, Civil Engineering - Computational Sciences
>>> > > > University of Houston, Department of Civil and Environmental
>>> Engineering
>>> > > > Houston, TX 77004
>>> > > > (512) 963-3262
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Justin Chang
>>> > > PhD Candidate, Civil Engineering - Computational Sciences
>>> > > University of Houston, Department of Civil and Environmental
>>> Engineering
>>> > > Houston, TX 77004
>>> > > (512) 963-3262
>>> >
>>> >
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/e71744fb/attachment.html>

From knepley at gmail.com  Tue Sep  1 04:02:33 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 Sep 2015 04:02:33 -0500
Subject: [petsc-users] Integrating TAO into SNES and TS
In-Reply-To: <CAP2=TMjdp3iKJ8943dHwKQmFg1cb9WOt29MgEgaiNTmMMcmoow@mail.gmail.com>
References: <CAP2=TMgF8fKyX0o--rRz+cw01=d5ws6iCZu=2PVq1qBJq50okA@mail.gmail.com>
	<16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov>
	<CAP2=TMhD41MWMf=n+Z7ZdDdNYvRVx91fL956AXSjtJb_vz01gg@mail.gmail.com>
	<0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov>
	<CAP2=TMh=NSnObLKwgR_ZJHZDqNe_c+VO4NPkSFnA3X+NLd0Amg@mail.gmail.com>
	<6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov>
	<CAP2=TMiZXce6FBKMRNBtnyn3Ug7cnMn_2ZhgR2547=OqpO8Nog@mail.gmail.com>
	<C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
	<CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>
	<CAMYG4G=MTDKpRrfnV+nBZvCGWNhTe-qtSkmVoCOF4J_uLS=e4A@mail.gmail.com>
	<CAP2=TMjdp3iKJ8943dHwKQmFg1cb9WOt29MgEgaiNTmMMcmoow@mail.gmail.com>
Message-ID: <CAMYG4GnvWpxNExr_VzkVt4q5UAEwMcHXMSYyahP9gxpP+-XRbw@mail.gmail.com>

On Tue, Sep 1, 2015 at 3:46 AM, Justin Chang <jychang48 at gmail.com> wrote:

> I would like to simultaneously enforce both discrete maximum principle and
> local mass/species balance. Because even if a locally conservative scheme
> like RT0 is used, as soon as these bounded constraints are applied, i lose
> the mass balance.
>

What I am saying is, can't you just add "linear equality constraints" as
more equations?

  Thanks,

    Matt


> On Tue, Sep 1, 2015 at 2:33 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Sep 1, 2015 at 3:11 AM, Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> Barry,
>>>
>>> That's good to know thanks.
>>>
>>> On a related note, is it possible for VI to one day include linear
>>> equality constraints?
>>>
>>
>> How are these different from just using more equations?
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Thanks,
>>> Justin
>>>
>>> On Mon, Aug 31, 2015 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>>
>>>> > On Aug 31, 2015, at 7:36 PM, Justin Chang <jychang48 at gmail.com>
>>>> wrote:
>>>> >
>>>> > Coming back to this,
>>>> >
>>>> > Say I now want to ensure the DMP for advection-diffusion equations.
>>>> The linear operator is now asymmetric and non-self-adjoint (assuming I do
>>>> something like SUPG or finite volume), meaning I cannot simply solve this
>>>> problem without any manipulation (e.g. normalizing the equations) using
>>>> TAO's optimization solvers. Does this statement also hold true for SNESVI?
>>>>
>>>>   SNESVI doesn't care about symmetry etc
>>>>
>>>> >
>>>> > Thanks,
>>>> > Justin
>>>> >
>>>> > On Fri, Apr 3, 2015 at 7:38 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>> >
>>>> > > On Apr 3, 2015, at 7:35 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>>> > >
>>>> > > I guess I will have to write my own code then :)
>>>> > >
>>>> > > I am not all that familiar with Variational Inequalities at the
>>>> moment, but if my Jacobian is symmetric and positive definite and I only
>>>> have lower and upper bounds, doesn't the problem simply reduce to that of a
>>>> convex optimization? That is, with SNES act as if it were Tao?
>>>> >
>>>> >   Yes, I think that is essentially correctly.
>>>> >
>>>> >   Barry
>>>> >
>>>> > >
>>>> > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>> > >
>>>> > >   Justin,
>>>> > >
>>>> > >    We haven't done anything with TS to handle variational
>>>> inequalities. So you can either write your own backward Euler (outside of
>>>> TS) that solves each time-step problem either as 1) an optimization problem
>>>> using Tao or 2) as a variational inequality using SNES.
>>>> > >
>>>> > >    More adventurously you could look at the TSTHETA code in TS
>>>> (which is a general form that includes Euler, Backward Euler and
>>>> Crank-Nicolson and see if you can add the constraints to the SNES problem
>>>> that is solved; in theory this is straightforward but it would require
>>>> understanding the current code (which Jed, of course, overwrote :-). I
>>>> think you should do this.
>>>> > >
>>>> > >   Barry
>>>> > >
>>>> > >
>>>> > > > On Apr 3, 2015, at 12:31 PM, Justin Chang <jchang27 at uh.edu>
>>>> wrote:
>>>> > > >
>>>> > > > I am solving the following anisotropic transient diffusion
>>>> equation subject to 0 bounds:
>>>> > > >
>>>> > > > du/dt = div[D*grad[u]] + f
>>>> > > >
>>>> > > > Where the dispersion tensor D(x) is symmetric and positive
>>>> definite. This formulation violates the discrete maximum principles so one
>>>> of the ways to ensure nonnegative concentrations is to employ convex
>>>> optimization. I am following the procedures in Nakshatrala and Valocchi
>>>> (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF.
>>>> > > >
>>>> > > > The Variational Inequality method works gives what I want for my
>>>> transient case, but what if I want to implement the Tao methodology in TS?
>>>> That is, what TS functions do I need to set up steps a) through e) for each
>>>> time step (also the Jacobian remains the same for all time steps so I would
>>>> only call this once). Normally I would just call TSSolve() and let the
>>>> libraries and functions do everything, but I would like to incorporate
>>>> TaoSolve into every time step.
>>>> > > >
>>>> > > > Thanks,
>>>> > > >
>>>> > > > --
>>>> > > > Justin Chang
>>>> > > > PhD Candidate, Civil Engineering - Computational Sciences
>>>> > > > University of Houston, Department of Civil and Environmental
>>>> Engineering
>>>> > > > Houston, TX 77004
>>>> > > > (512) 963-3262
>>>> > > >
>>>> > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>> > > >
>>>> > > >   An alternative approach is for you to solve it as a (non)linear
>>>> variational inequality. See src/snes/examples/tutorials/ex9.c
>>>> > > >
>>>> > > >   How you should proceed depends on your long term goal. What
>>>> problem do you really want to solve? Is it really a linear time dependent
>>>> problem with 0 bounds on U? Can the problem always be represented as an
>>>> optimization problem easily? What are  and what will be the properties of
>>>> K? For example if K is positive definite then likely the bounds will remain
>>>> try without explicitly providing the constraints.
>>>> > > >
>>>> > > >   Barry
>>>> > > >
>>>> > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang <jchang27 at uh.edu>
>>>> wrote:
>>>> > > > >
>>>> > > > > Hi everyone,
>>>> > > > >
>>>> > > > > I have a two part question regarding the integration of the
>>>> following optimization problem
>>>> > > > >
>>>> > > > > min 1/2 u^T*K*u + u^T*f
>>>> > > > > S.T. u >= 0
>>>> > > > >
>>>> > > > > into SNES and TS
>>>> > > > >
>>>> > > > > 1) For SNES, assuming I am working with a linear FE equation, I
>>>> have the following algorithm/steps for solving my problem
>>>> > > > >
>>>> > > > > a) Set an initial guess x
>>>> > > > > b) Obtain residual r and jacobian A through functions
>>>> SNESComputeFunction() and SNESComputeJacobian() respectively
>>>> > > > > c) Form vector b = r - A*x
>>>> > > > > d) Set Hessian equal to A, gradient to A*x, objective function
>>>> value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector
>>>> > > > > e) Call TaoSolve
>>>> > > > >
>>>> > > > > This works well at the moment, but my question is there a more
>>>> "efficient" way of doing this? Because with my current setup, I am making a
>>>> rather bold assumption that my problem would converge in one SNES iteration
>>>> without the bounded constraints and does not have any unexpected
>>>> nonlinearities.
>>>> > > > >
>>>> > > > > 2) How would I go about doing the above for time-stepping
>>>> problems? At each time step, I want to solve a convex optimization subject
>>>> to the lower bounds constraint. I plan on using backward euler and my
>>>> resulting jacobian should still be compatible with the above optimization
>>>> problem.
>>>> > > > >
>>>> > > > > Thanks,
>>>> > > > >
>>>> > > > > --
>>>> > > > > Justin Chang
>>>> > > > > PhD Candidate, Civil Engineering - Computational Sciences
>>>> > > > > University of Houston, Department of Civil and Environmental
>>>> Engineering
>>>> > > > > Houston, TX 77004
>>>> > > > > (512) 963-3262
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > --
>>>> > > > Justin Chang
>>>> > > > PhD Candidate, Civil Engineering - Computational Sciences
>>>> > > > University of Houston, Department of Civil and Environmental
>>>> Engineering
>>>> > > > Houston, TX 77004
>>>> > > > (512) 963-3262
>>>> > >
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Justin Chang
>>>> > > PhD Candidate, Civil Engineering - Computational Sciences
>>>> > > University of Houston, Department of Civil and Environmental
>>>> Engineering
>>>> > > Houston, TX 77004
>>>> > > (512) 963-3262
>>>> >
>>>> >
>>>>
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/a5d5266e/attachment-0001.html>

From filippo.leonardi at sam.math.ethz.ch  Tue Sep  1 04:12:23 2015
From: filippo.leonardi at sam.math.ethz.ch (Filippo Leonardi)
Date: Tue, 1 Sep 2015 11:12:23 +0200
Subject: [petsc-users] 2D Poisson on nonuniform meshes
In-Reply-To: <CAMYG4Gm1KATfJAEJeZT_AGpwyS+Vd5RYLfXpda=JMANtd_ANrQ@mail.gmail.com>
References: <3634751.Hfn5OcL7V3@besikowitch-iii>
	<CAMYG4Gm1KATfJAEJeZT_AGpwyS+Vd5RYLfXpda=JMANtd_ANrQ@mail.gmail.com>
Message-ID: <5342199.ihN3W0jbJW@besikowitch-iii>

Thanks Matt,

I use the "natural scaling"  of the Laplacian Lapl * x  = b. The grid is *still* uniform.

I give you a couple of results:

1 ) This is e.g. gmres + gamg + 4 levels. As you can see it converges after 2 iterations which is 
totally strange. The solution is incorrect.

2 ) This is mg + galerkin matrices + levels. Result is incorrect

3 ) Default: works fine

4) LU: works fine

I remove the constant nullspace as usual.

I use the usual setcomputeoperator business and compute rhs.

Instead, if I scale to Lapl * dx * dy * x = b * dx * dy (things work fine also with MG). This is fine 
with uniform grid but  I do not know how to do that properly on nonuniform.

Let me know if you need code pieces or other details.

1)

 0 KSP Residual norm 1.793027353791e-05  


2)

 0 KSP Residual norm 1.010294060491e-05  


0 KSP Residual norm 2.990548680394e-08  


Linear solve converged due to CONVERGED_ITS iterations 1 


On Tuesday 01 September 2015 03:40:14 Matthew Knepley wrote:
> On Tue, Sep 1, 2015 at 2:22 AM, Filippo Leonardi <
> 
> filippo.leonardi at sam.math.ethz.ch> wrote:
> > Dear PETSc Users,
> > 
> > 
> > 
> > I want to use multigrid to solve uniform (just Laplace) poisson in 2D/3D
> > on cartesian, non-uniform meshes with a standard 5 (7)-points stencil FD.
> > 
> > 
> > 
> > I always scaled my Poisson matrix like in the doc examples, i.e.
> > multiplying by dx*dy (so that in ComputeRHS I need to scale b, in A*x = b,
> > as well). This always worked properly with both MG/GAMG and with galerkin
> > matrices.
> > 
> > 
> > 
> > Now I'd like to use non-uniform meshes, therefore the scaling is
> > non-uniform. However I cannot get my matrices to scale properly with any
> > sort of multigrid.
> > 
> > 
> > 
> > One would think that without scaling, i.e. solving the original system, at
> > least MG+galerkin or GAMG should work anyways provided the matrix A and b
> > are consistent.
> > 
> > 
> > 
> > I tried without boundaries (i.e. torus), so this is not the problem.
> 
> The point of that scaling is to make the boundary values the same size as
> the residuals. If you have
> no boundary values, the problem is scale invariant since its linear. It
> would be nice to scale it so everything
> is about size 1.
> 
> When asking about convergence always run with -ksp_view
> -ksp_monitor_true_residual -ksp_converged_reason
> 
>   Thanks,
> 
>      Matt
> 
> > Anyone did/knows how to do this properly?
> > 
> > 
> > 
> > Thanks,
> > 
> > F

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/da3abb44/attachment-0001.html>

From jychang48 at gmail.com  Tue Sep  1 04:15:05 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 1 Sep 2015 03:15:05 -0600
Subject: [petsc-users] Integrating TAO into SNES and TS
In-Reply-To: <CAMYG4GnvWpxNExr_VzkVt4q5UAEwMcHXMSYyahP9gxpP+-XRbw@mail.gmail.com>
References: <CAP2=TMgF8fKyX0o--rRz+cw01=d5ws6iCZu=2PVq1qBJq50okA@mail.gmail.com>
	<16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov>
	<CAP2=TMhD41MWMf=n+Z7ZdDdNYvRVx91fL956AXSjtJb_vz01gg@mail.gmail.com>
	<0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov>
	<CAP2=TMh=NSnObLKwgR_ZJHZDqNe_c+VO4NPkSFnA3X+NLd0Amg@mail.gmail.com>
	<6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov>
	<CAP2=TMiZXce6FBKMRNBtnyn3Ug7cnMn_2ZhgR2547=OqpO8Nog@mail.gmail.com>
	<C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
	<CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>
	<CAMYG4G=MTDKpRrfnV+nBZvCGWNhTe-qtSkmVoCOF4J_uLS=e4A@mail.gmail.com>
	<CAP2=TMjdp3iKJ8943dHwKQmFg1cb9WOt29MgEgaiNTmMMcmoow@mail.gmail.com>
	<CAMYG4GnvWpxNExr_VzkVt4q5UAEwMcHXMSYyahP9gxpP+-XRbw@mail.gmail.com>
Message-ID: <CAP2=TMiJUz6yM5vMjThMTgZ6CmRsS+da6dU+4FuJq8qk6c+fdw@mail.gmail.com>

But if I add those linear equality constraint equations to my original
problem, would they not be satisfied anyway? Say I add this to my weak form:

Ax = b

But once i subject x to some bounded constraints, Ax != b. Unless I add
some sort of penalty where extra weighting is added to this property...

On Tue, Sep 1, 2015 at 3:02 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Sep 1, 2015 at 3:46 AM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> I would like to simultaneously enforce both discrete maximum principle
>> and local mass/species balance. Because even if a locally conservative
>> scheme like RT0 is used, as soon as these bounded constraints are applied,
>> i lose the mass balance.
>>
>
> What I am saying is, can't you just add "linear equality constraints" as
> more equations?
>
>   Thanks,
>
>     Matt
>
>
>> On Tue, Sep 1, 2015 at 2:33 AM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Tue, Sep 1, 2015 at 3:11 AM, Justin Chang <jychang48 at gmail.com>
>>> wrote:
>>>
>>>> Barry,
>>>>
>>>> That's good to know thanks.
>>>>
>>>> On a related note, is it possible for VI to one day include linear
>>>> equality constraints?
>>>>
>>>
>>> How are these different from just using more equations?
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> Thanks,
>>>> Justin
>>>>
>>>> On Mon, Aug 31, 2015 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>>
>>>>>
>>>>> > On Aug 31, 2015, at 7:36 PM, Justin Chang <jychang48 at gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Coming back to this,
>>>>> >
>>>>> > Say I now want to ensure the DMP for advection-diffusion equations.
>>>>> The linear operator is now asymmetric and non-self-adjoint (assuming I do
>>>>> something like SUPG or finite volume), meaning I cannot simply solve this
>>>>> problem without any manipulation (e.g. normalizing the equations) using
>>>>> TAO's optimization solvers. Does this statement also hold true for SNESVI?
>>>>>
>>>>>   SNESVI doesn't care about symmetry etc
>>>>>
>>>>> >
>>>>> > Thanks,
>>>>> > Justin
>>>>> >
>>>>> > On Fri, Apr 3, 2015 at 7:38 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>> wrote:
>>>>> >
>>>>> > > On Apr 3, 2015, at 7:35 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>>>> > >
>>>>> > > I guess I will have to write my own code then :)
>>>>> > >
>>>>> > > I am not all that familiar with Variational Inequalities at the
>>>>> moment, but if my Jacobian is symmetric and positive definite and I only
>>>>> have lower and upper bounds, doesn't the problem simply reduce to that of a
>>>>> convex optimization? That is, with SNES act as if it were Tao?
>>>>> >
>>>>> >   Yes, I think that is essentially correctly.
>>>>> >
>>>>> >   Barry
>>>>> >
>>>>> > >
>>>>> > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>> wrote:
>>>>> > >
>>>>> > >   Justin,
>>>>> > >
>>>>> > >    We haven't done anything with TS to handle variational
>>>>> inequalities. So you can either write your own backward Euler (outside of
>>>>> TS) that solves each time-step problem either as 1) an optimization problem
>>>>> using Tao or 2) as a variational inequality using SNES.
>>>>> > >
>>>>> > >    More adventurously you could look at the TSTHETA code in TS
>>>>> (which is a general form that includes Euler, Backward Euler and
>>>>> Crank-Nicolson and see if you can add the constraints to the SNES problem
>>>>> that is solved; in theory this is straightforward but it would require
>>>>> understanding the current code (which Jed, of course, overwrote :-). I
>>>>> think you should do this.
>>>>> > >
>>>>> > >   Barry
>>>>> > >
>>>>> > >
>>>>> > > > On Apr 3, 2015, at 12:31 PM, Justin Chang <jchang27 at uh.edu>
>>>>> wrote:
>>>>> > > >
>>>>> > > > I am solving the following anisotropic transient diffusion
>>>>> equation subject to 0 bounds:
>>>>> > > >
>>>>> > > > du/dt = div[D*grad[u]] + f
>>>>> > > >
>>>>> > > > Where the dispersion tensor D(x) is symmetric and positive
>>>>> definite. This formulation violates the discrete maximum principles so one
>>>>> of the ways to ensure nonnegative concentrations is to employ convex
>>>>> optimization. I am following the procedures in Nakshatrala and Valocchi
>>>>> (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF.
>>>>> > > >
>>>>> > > > The Variational Inequality method works gives what I want for my
>>>>> transient case, but what if I want to implement the Tao methodology in TS?
>>>>> That is, what TS functions do I need to set up steps a) through e) for each
>>>>> time step (also the Jacobian remains the same for all time steps so I would
>>>>> only call this once). Normally I would just call TSSolve() and let the
>>>>> libraries and functions do everything, but I would like to incorporate
>>>>> TaoSolve into every time step.
>>>>> > > >
>>>>> > > > Thanks,
>>>>> > > >
>>>>> > > > --
>>>>> > > > Justin Chang
>>>>> > > > PhD Candidate, Civil Engineering - Computational Sciences
>>>>> > > > University of Houston, Department of Civil and Environmental
>>>>> Engineering
>>>>> > > > Houston, TX 77004
>>>>> > > > (512) 963-3262
>>>>> > > >
>>>>> > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>> wrote:
>>>>> > > >
>>>>> > > >   An alternative approach is for you to solve it as a
>>>>> (non)linear variational inequality. See src/snes/examples/tutorials/ex9.c
>>>>> > > >
>>>>> > > >   How you should proceed depends on your long term goal. What
>>>>> problem do you really want to solve? Is it really a linear time dependent
>>>>> problem with 0 bounds on U? Can the problem always be represented as an
>>>>> optimization problem easily? What are  and what will be the properties of
>>>>> K? For example if K is positive definite then likely the bounds will remain
>>>>> try without explicitly providing the constraints.
>>>>> > > >
>>>>> > > >   Barry
>>>>> > > >
>>>>> > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang <jchang27 at uh.edu>
>>>>> wrote:
>>>>> > > > >
>>>>> > > > > Hi everyone,
>>>>> > > > >
>>>>> > > > > I have a two part question regarding the integration of the
>>>>> following optimization problem
>>>>> > > > >
>>>>> > > > > min 1/2 u^T*K*u + u^T*f
>>>>> > > > > S.T. u >= 0
>>>>> > > > >
>>>>> > > > > into SNES and TS
>>>>> > > > >
>>>>> > > > > 1) For SNES, assuming I am working with a linear FE equation,
>>>>> I have the following algorithm/steps for solving my problem
>>>>> > > > >
>>>>> > > > > a) Set an initial guess x
>>>>> > > > > b) Obtain residual r and jacobian A through functions
>>>>> SNESComputeFunction() and SNESComputeJacobian() respectively
>>>>> > > > > c) Form vector b = r - A*x
>>>>> > > > > d) Set Hessian equal to A, gradient to A*x, objective function
>>>>> value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector
>>>>> > > > > e) Call TaoSolve
>>>>> > > > >
>>>>> > > > > This works well at the moment, but my question is there a more
>>>>> "efficient" way of doing this? Because with my current setup, I am making a
>>>>> rather bold assumption that my problem would converge in one SNES iteration
>>>>> without the bounded constraints and does not have any unexpected
>>>>> nonlinearities.
>>>>> > > > >
>>>>> > > > > 2) How would I go about doing the above for time-stepping
>>>>> problems? At each time step, I want to solve a convex optimization subject
>>>>> to the lower bounds constraint. I plan on using backward euler and my
>>>>> resulting jacobian should still be compatible with the above optimization
>>>>> problem.
>>>>> > > > >
>>>>> > > > > Thanks,
>>>>> > > > >
>>>>> > > > > --
>>>>> > > > > Justin Chang
>>>>> > > > > PhD Candidate, Civil Engineering - Computational Sciences
>>>>> > > > > University of Houston, Department of Civil and Environmental
>>>>> Engineering
>>>>> > > > > Houston, TX 77004
>>>>> > > > > (512) 963-3262
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > --
>>>>> > > > Justin Chang
>>>>> > > > PhD Candidate, Civil Engineering - Computational Sciences
>>>>> > > > University of Houston, Department of Civil and Environmental
>>>>> Engineering
>>>>> > > > Houston, TX 77004
>>>>> > > > (512) 963-3262
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > Justin Chang
>>>>> > > PhD Candidate, Civil Engineering - Computational Sciences
>>>>> > > University of Houston, Department of Civil and Environmental
>>>>> Engineering
>>>>> > > Houston, TX 77004
>>>>> > > (512) 963-3262
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/9e253e5e/attachment.html>

From knepley at gmail.com  Tue Sep  1 04:15:50 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 Sep 2015 04:15:50 -0500
Subject: [petsc-users] 2D Poisson on nonuniform meshes
In-Reply-To: <5342199.ihN3W0jbJW@besikowitch-iii>
References: <3634751.Hfn5OcL7V3@besikowitch-iii>
	<CAMYG4Gm1KATfJAEJeZT_AGpwyS+Vd5RYLfXpda=JMANtd_ANrQ@mail.gmail.com>
	<5342199.ihN3W0jbJW@besikowitch-iii>
Message-ID: <CAMYG4GnvdUhFhBEpsFrGb7vtH24us3TU-8LMuT4WWUT=z-6evw@mail.gmail.com>

On Tue, Sep 1, 2015 at 4:12 AM, Filippo Leonardi <
filippo.leonardi at sam.math.ethz.ch> wrote:

> Thanks Matt,
>
>
>
> I use the "natural scaling" of the Laplacian Lapl * x = b. The grid is
> *still* uniform.
>
>
>
> I give you a couple of results:
>
>
>
> 1 ) This is e.g. gmres + gamg + 4 levels. As you can see it converges
> after 2 iterations which is totally strange. The solution is incorrect.
>
>
>
> 2 ) This is mg + galerkin matrices + levels. Result is incorrect
>
>
>
> 3 ) Default: works fine
>
>
>
> 4) LU: works fine
>
>
>
> I remove the constant nullspace as usual.
>

If you have a constant null space, how does 4) work? It should fail with a
zero pivot since
the matrix is singular.

   Matt


> I use the usual setcomputeoperator business and compute rhs.
>
>
>
> Instead, if I scale to Lapl * dx * dy * x = b * dx * dy (things work fine
> also with MG). This is fine with uniform grid but I do not know how to do
> that properly on nonuniform.
>
>
>
> Let me know if you need code pieces or other details.
>
>
>
> 1)
>
>
>
>  0 KSP Residual norm 1.793027353791e-05
>  0 KSP preconditioned resid norm 1.793027353791e-05 true resid norm
> 7.912994695029e-03 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP Residual norm 2.662212708666e-20
>  1 KSP preconditioned resid norm 2.662212708666e-20 true resid norm
> 3.120963188535e-02 ||r(i)||/||b|| 3.944098674167e+00
> Linear solve converged due to CONVERGED_RTOL iterations 1
> KSP Object: 1 MPI processes
>  type: gmres
>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>    GMRES: happy breakdown tolerance 1e-30
>  maximum iterations=10000, initial guess is zero
>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>  left preconditioning
>  using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>  type: gamg
>    MG: type is MULTIPLICATIVE, levels=4 cycles=v
>      Cycles per PCApply=1
>      Using Galerkin computed coarse grid matrices
>      GAMG specific options
>        Threshold for dropping small values from graph 0
>        AGG specific options
>          Symmetric graph false
>  Coarse grid solver -- level -------------------------------
>    KSP Object:    (mg_coarse_)     1 MPI processes
>      type: gmres
>        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>        GMRES: happy breakdown tolerance 1e-30
>      maximum iterations=1, initial guess is zero
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object:    (mg_coarse_)     1 MPI processes
>      type: bjacobi
>        block Jacobi: number of blocks = 1
>        Local solve is same for all blocks, in the following KSP and PC
> objects:
>        KSP Object:        (mg_coarse_sub_)         1 MPI processes
>          type: preonly
>          maximum iterations=1, initial guess is zero
>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>          left preconditioning
>          using NONE norm type for convergence test
>        PC Object:        (mg_coarse_sub_)         1 MPI processes
>          type: lu
>            LU: out-of-place factorization
>            tolerance for zero pivot 2.22045e-14
>            using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>            matrix ordering: nd
>            factor fill ratio given 5, needed 1.10953
>              Factored matrix follows:
>                Mat Object:                 1 MPI processes
>                  type: seqaij
>                  rows=33, cols=33
>                  package used to perform factorization: petsc
>                  total: nonzeros=1013, allocated nonzeros=1013
>                  total number of mallocs used during MatSetValues calls =0
>                    using I-node routines: found 21 nodes, limit used is 5
>          linear system matrix = precond matrix:
>          Mat Object:           1 MPI processes
>            type: seqaij
>            rows=33, cols=33
>            total: nonzeros=913, allocated nonzeros=913
>            total number of mallocs used during MatSetValues calls =0
>              not using I-node routines
>      linear system matrix = precond matrix:
>      Mat Object:       1 MPI processes
>        type: seqaij
>        rows=33, cols=33
>        total: nonzeros=913, allocated nonzeros=913
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node routines
>  Down solver (pre-smoother) on level 1 -------------------------------
>    KSP Object:    (mg_levels_1_)     1 MPI processes
>      type: chebyshev
>        Chebyshev: eigenvalue estimates:  min = -1.79769e+307, max = -inf
>        Chebyshev: eigenvalues estimated using gmres with translations  [0
> 0.1; 0 1.1]
>        KSP Object:        (mg_levels_1_esteig_)         1 MPI processes
>          type: gmres
>            GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            GMRES: happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>          left preconditioning
>          using NONE norm type for convergence test
>      maximum iterations=2
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using nonzero initial guess
>      using NONE norm type for convergence test
>    PC Object:    (mg_levels_1_)     1 MPI processes
>      type: sor
>        SOR: type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1
>      linear system matrix = precond matrix:
>      Mat Object:       1 MPI processes
>        type: seqaij
>        rows=401, cols=401
>        total: nonzeros=9777, allocated nonzeros=9777
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 2 -------------------------------
>    KSP Object:    (mg_levels_2_)     1 MPI processes
>      type: chebyshev
>        Chebyshev: eigenvalue estimates:  min = -1.79769e+307, max = -inf
>        Chebyshev: eigenvalues estimated using gmres with translations  [0
> 0.1; 0 1.1]
>        KSP Object:        (mg_levels_2_esteig_)         1 MPI processes
>          type: gmres
>            GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            GMRES: happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>          left preconditioning
>          using NONE norm type for convergence test
>      maximum iterations=2
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using nonzero initial guess
>      using NONE norm type for convergence test
>    PC Object:    (mg_levels_2_)     1 MPI processes
>      type: sor
>        SOR: type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1
>      linear system matrix = precond matrix:
>      Mat Object:       1 MPI processes
>        type: seqaij
>        rows=2296, cols=2296
>        total: nonzeros=25038, allocated nonzeros=25038
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 3 -------------------------------
>    KSP Object:    (mg_levels_3_)     1 MPI processes
>      type: chebyshev
>        Chebyshev: eigenvalue estimates:  min = -1.79769e+307, max = -inf
>        Chebyshev: eigenvalues estimated using gmres with translations  [0
> 0.1; 0 1.1]
>        KSP Object:        (mg_levels_3_esteig_)         1 MPI processes
>          type: gmres
>            GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            GMRES: happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>          left preconditioning
>          using NONE norm type for convergence test
>      maximum iterations=2
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using nonzero initial guess
>      using NONE norm type for convergence test
>    PC Object:    (mg_levels_3_)     1 MPI processes
>      type: sor
>        SOR: type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1
>      linear system matrix = precond matrix:
>      Mat Object:       1 MPI processes
>        type: seqaij
>        rows=16384, cols=16384
>        total: nonzeros=81920, allocated nonzeros=81920
>        total number of mallocs used during MatSetValues calls =0
>          has attached null space
>          not using I-node routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  linear system matrix = precond matrix:
>  Mat Object:   1 MPI processes
>    type: seqaij
>    rows=16384, cols=16384
>    total: nonzeros=81920, allocated nonzeros=81920
>    total number of mallocs used during MatSetValues calls =0
>      has attached null space
>      not using I-node routines
>
>
>
> 2)
>
>
>
>  0 KSP Residual norm 1.010294060491e-05
>  0 KSP preconditioned resid norm 1.010294060491e-05 true resid norm
> 1.086379257868e-03 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP Residual norm 8.803303876758e-20
>  1 KSP preconditioned resid norm 8.803303876758e-20 true resid norm
> 2.101751936637e-03 ||r(i)||/||b|| 1.934639235254e+00
> Linear solve converged due to CONVERGED_RTOL iterations 1
> KSP Object: 1 MPI processes
>  type: gmres
>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>    GMRES: happy breakdown tolerance 1e-30
>  maximum iterations=10000, initial guess is zero
>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>  left preconditioning
>  using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>  type: mg
>    MG: type is MULTIPLICATIVE, levels=4 cycles=v
>      Cycles per PCApply=1
>      Using Galerkin computed coarse grid matrices
>  Coarse grid solver -- level -------------------------------
>    KSP Object:    (mg_coarse_)     1 MPI processes
>      type: preonly
>      maximum iterations=1, initial guess is zero
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object:    (mg_coarse_)     1 MPI processes
>      type: lu
>        LU: out-of-place factorization
>        tolerance for zero pivot 2.22045e-14
>        using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>        matrix ordering: nd
>        factor fill ratio given 5, needed 4.83507
>          Factored matrix follows:
>            Mat Object:             1 MPI processes
>              type: seqaij
>              rows=256, cols=256
>              package used to perform factorization: petsc
>              total: nonzeros=11140, allocated nonzeros=11140
>              total number of mallocs used during MatSetValues calls =0
>                not using I-node routines
>      linear system matrix = precond matrix:
>      Mat Object:       1 MPI processes
>        type: seqaij
>        rows=256, cols=256
>        total: nonzeros=2304, allocated nonzeros=2304
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node routines
>  Down solver (pre-smoother) on level 1 -------------------------------
>    KSP Object:    (mg_levels_1_)     1 MPI processes
>      type: chebyshev
>        Chebyshev: eigenvalue estimates:  min = -1.79769e+307, max = -inf
>        Chebyshev: eigenvalues estimated using gmres with translations  [0
> 0.1; 0 1.1]
>        KSP Object:        (mg_levels_1_esteig_)         1 MPI processes
>          type: gmres
>            GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            GMRES: happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>          left preconditioning
>          using NONE norm type for convergence test
>      maximum iterations=2
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using nonzero initial guess
>      using NONE norm type for convergence test
>    PC Object:    (mg_levels_1_)     1 MPI processes
>      type: sor
>        SOR: type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1
>      linear system matrix = precond matrix:
>      Mat Object:       1 MPI processes
>        type: seqaij
>        rows=1024, cols=1024
>        total: nonzeros=9216, allocated nonzeros=9216
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 2 -------------------------------
>    KSP Object:    (mg_levels_2_)     1 MPI processes
>      type: chebyshev
>        Chebyshev: eigenvalue estimates:  min = -1.79769e+307, max = -inf
>        Chebyshev: eigenvalues estimated using gmres with translations  [0
> 0.1; 0 1.1]
>        KSP Object:        (mg_levels_2_esteig_)         1 MPI processes
>          type: gmres
>            GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            GMRES: happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>          left preconditioning
>          using NONE norm type for convergence test
>      maximum iterations=2
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using nonzero initial guess
>      using NONE norm type for convergence test
>    PC Object:    (mg_levels_2_)     1 MPI processes
>      type: sor
>        SOR: type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1
>      linear system matrix = precond matrix:
>      Mat Object:       1 MPI processes
>        type: seqaij
>        rows=4096, cols=4096
>        total: nonzeros=36864, allocated nonzeros=36864
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 3 -------------------------------
>    KSP Object:    (mg_levels_3_)     1 MPI processes
>      type: chebyshev
>        Chebyshev: eigenvalue estimates:  min = -1.79769e+307, max = -inf
>        Chebyshev: eigenvalues estimated using gmres with translations  [0
> 0.1; 0 1.1]
>        KSP Object:        (mg_levels_3_esteig_)         1 MPI processes
>          type: gmres
>            GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            GMRES: happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>          left preconditioning
>          using NONE norm type for convergence test
>      maximum iterations=2
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using nonzero initial guess
>      using NONE norm type for convergence test
>    PC Object:    (mg_levels_3_)     1 MPI processes
>      type: sor
>        SOR: type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1
>      linear system matrix = precond matrix:
>      Mat Object:       1 MPI processes
>        type: seqaij
>        rows=16384, cols=16384
>        total: nonzeros=81920, allocated nonzeros=81920
>        total number of mallocs used during MatSetValues calls =0
>          has attached null space
>          not using I-node routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  linear system matrix = precond matrix:
>  Mat Object:   1 MPI processes
>    type: seqaij
>    rows=16384, cols=16384
>    total: nonzeros=81920, allocated nonzeros=81920
>    total number of mallocs used during MatSetValues calls =0
>      has attached null space
>      not using I-node routines
>
> 3)
>
>
>
> 0 KSP Residual norm 2.990548680394e-08
>  0 KSP preconditioned resid norm 2.990548680394e-08 true resid norm
> 2.966915112865e-04 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP Residual norm 2.368703753809e-08
>  1 KSP preconditioned resid norm 2.368703753809e-08 true resid norm
> 3.109667150576e-04 ||r(i)||/||b|| 1.048114634993e+00
>  2 KSP Residual norm 2.236515515253e-08
>  2 KSP preconditioned resid norm 2.236515515253e-08 true resid norm
> 2.655771471955e-04 ||r(i)||/||b|| 8.951289035670e-01
>  3 KSP Residual norm 1.957630940223e-08
>  3 KSP preconditioned resid norm 1.957630940223e-08 true resid norm
> 2.158250768646e-04 ||r(i)||/||b|| 7.274393390250e-01
>  4 KSP Residual norm 1.771416739221e-08
>  4 KSP preconditioned resid norm 1.771416739221e-08 true resid norm
> 2.075204223802e-04 ||r(i)||/||b|| 6.994484657830e-01
>  5 KSP Residual norm 1.617056152936e-08
>  5 KSP preconditioned resid norm 1.617056152936e-08 true resid norm
> 1.908090487285e-04 ||r(i)||/||b|| 6.431227098516e-01
>  6 KSP Residual norm 1.467170521607e-08
>  6 KSP preconditioned resid norm 1.467170521607e-08 true resid norm
> 1.683698119419e-04 ||r(i)||/||b|| 5.674911668750e-01
>  7 KSP Residual norm 1.321526248809e-08
>  7 KSP preconditioned resid norm 1.321526248809e-08 true resid norm
> 1.473617884651e-04 ||r(i)||/||b|| 4.966835344432e-01
>  8 KSP Residual norm 1.174828665876e-08
>  8 KSP preconditioned resid norm 1.174828665876e-08 true resid norm
> 1.458902362707e-04 ||r(i)||/||b|| 4.917236615167e-01
>  9 KSP Residual norm 1.012151625603e-08
>  9 KSP preconditioned resid norm 1.012151625603e-08 true resid norm
> 1.303866699420e-04 ||r(i)||/||b|| 4.394688253015e-01
> 10 KSP Residual norm 7.876439970777e-09
> 10 KSP preconditioned resid norm 7.876439970777e-09 true resid norm
> 1.162414266035e-04 ||r(i)||/||b|| 3.917922224990e-01
> 11 KSP Residual norm 4.981821461221e-09
> 11 KSP preconditioned resid norm 4.981821461221e-09 true resid norm
> 9.075981784793e-05 ||r(i)||/||b|| 3.059063518683e-01
> 12 KSP Residual norm 2.426217812448e-09
> 12 KSP preconditioned resid norm 2.426217812448e-09 true resid norm
> 5.145334699067e-05 ||r(i)||/||b|| 1.734237247556e-01
> 13 KSP Residual norm 1.049060114422e-09
> 13 KSP preconditioned resid norm 1.049060114422e-09 true resid norm
> 2.107527819651e-05 ||r(i)||/||b|| 7.103431475044e-02
> 14 KSP Residual norm 5.887222551677e-10
> 14 KSP preconditioned resid norm 5.887222551677e-10 true resid norm
> 9.047187295472e-06 ||r(i)||/||b|| 3.049358323817e-02
> 15 KSP Residual norm 4.536407095834e-10
> 15 KSP preconditioned resid norm 4.536407095834e-10 true resid norm
> 6.032647097712e-06 ||r(i)||/||b|| 2.033306268708e-02
> 16 KSP Residual norm 3.564018350025e-10
> 16 KSP preconditioned resid norm 3.564018350025e-10 true resid norm
> 5.116056274149e-06 ||r(i)||/||b|| 1.724368941991e-02
> 17 KSP Residual norm 2.744968239957e-10
> 17 KSP preconditioned resid norm 2.744968239957e-10 true resid norm
> 3.648248676660e-06 ||r(i)||/||b|| 1.229643767306e-02
> 18 KSP Residual norm 2.281847233955e-10
> 18 KSP preconditioned resid norm 2.281847233955e-10 true resid norm
> 2.900148270786e-06 ||r(i)||/||b|| 9.774962074955e-03
> 19 KSP Residual norm 1.881670741813e-10
> 19 KSP preconditioned resid norm 1.881670741813e-10 true resid norm
> 2.473850917169e-06 ||r(i)||/||b|| 8.338125032436e-03
> 20 KSP Residual norm 1.581922522755e-10
> 20 KSP preconditioned resid norm 1.581922522755e-10 true resid norm
> 1.965254213304e-06 ||r(i)||/||b|| 6.623897680059e-03
> 21 KSP Residual norm 1.349952347592e-10
> 21 KSP preconditioned resid norm 1.349952347592e-10 true resid norm
> 1.709953736235e-06 ||r(i)||/||b|| 5.763406336838e-03
> 22 KSP Residual norm 1.148754892680e-10
> 22 KSP preconditioned resid norm 1.148754892680e-10 true resid norm
> 1.452723184020e-06 ||r(i)||/||b|| 4.896409667135e-03
> 23 KSP Residual norm 9.889008860663e-11
> 23 KSP preconditioned resid norm 9.889008860663e-11 true resid norm
> 1.259317356950e-06 ||r(i)||/||b|| 4.244534504846e-03
> 24 KSP Residual norm 8.468070834423e-11
> 24 KSP preconditioned resid norm 8.468070834423e-11 true resid norm
> 1.097968496396e-06 ||r(i)||/||b|| 3.700707484468e-03
> 25 KSP Residual norm 7.340678871498e-11
> 25 KSP preconditioned resid norm 7.340678871498e-11 true resid norm
> 9.388293102378e-07 ||r(i)||/||b|| 3.164328181036e-03
> 26 KSP Residual norm 6.452497596580e-11
> 26 KSP preconditioned resid norm 6.452497596580e-11 true resid norm
> 7.907509627807e-07 ||r(i)||/||b|| 2.665229481464e-03
> 27 KSP Residual norm 5.817349095331e-11
> 27 KSP preconditioned resid norm 5.817349095331e-11 true resid norm
> 6.712549222809e-07 ||r(i)||/||b|| 2.262467568992e-03
> 28 KSP Residual norm 5.382715584394e-11
> 28 KSP preconditioned resid norm 5.382715584394e-11 true resid norm
> 5.883671521302e-07 ||r(i)||/||b|| 1.983093987351e-03
> 29 KSP Residual norm 5.072666199584e-11
> 29 KSP preconditioned resid norm 5.072666199584e-11 true resid norm
> 5.417512216577e-07 ||r(i)||/||b|| 1.825974795533e-03
> 30 KSP Residual norm 4.806650672230e-11
> 30 KSP preconditioned resid norm 4.806650672230e-11 true resid norm
> 5.151866485899e-07 ||r(i)||/||b|| 1.736438789084e-03
> 31 KSP Residual norm 4.675001898732e-11
> 31 KSP preconditioned resid norm 4.675001898732e-11 true resid norm
> 4.825947510614e-07 ||r(i)||/||b|| 1.626587659919e-03
> 32 KSP Residual norm 4.530536181592e-11
> 32 KSP preconditioned resid norm 4.530536181592e-11 true resid norm
> 4.648650267512e-07 ||r(i)||/||b|| 1.566829548764e-03
> 33 KSP Residual norm 4.357157235509e-11
> 33 KSP preconditioned resid norm 4.357157235509e-11 true resid norm
> 4.567209908707e-07 ||r(i)||/||b|| 1.539380041209e-03
> 34 KSP Residual norm 4.135715715343e-11
> 34 KSP preconditioned resid norm 4.135715715343e-11 true resid norm
> 4.469703503011e-07 ||r(i)||/||b|| 1.506515465721e-03
> 35 KSP Residual norm 3.851871986267e-11
> 35 KSP preconditioned resid norm 3.851871986267e-11 true resid norm
> 4.364097171314e-07 ||r(i)||/||b|| 1.470920806730e-03
> 36 KSP Residual norm 3.526035391634e-11
> 36 KSP preconditioned resid norm 3.526035391634e-11 true resid norm
> 4.065382561286e-07 ||r(i)||/||b|| 1.370238920439e-03
> 37 KSP Residual norm 3.144895012756e-11
> 37 KSP preconditioned resid norm 3.144895012756e-11 true resid norm
> 3.810168560499e-07 ||r(i)||/||b|| 1.284218932985e-03
> 38 KSP Residual norm 2.735239124106e-11
> 38 KSP preconditioned resid norm 2.735239124106e-11 true resid norm
> 3.407338934429e-07 ||r(i)||/||b|| 1.148445036278e-03
> 39 KSP Residual norm 2.306734554159e-11
> 39 KSP preconditioned resid norm 2.306734554159e-11 true resid norm
> 2.940484127641e-07 ||r(i)||/||b|| 9.910914251948e-04
> 40 KSP Residual norm 1.924842110922e-11
> 40 KSP preconditioned resid norm 1.924842110922e-11 true resid norm
> 2.443693779909e-07 ||r(i)||/||b|| 8.236480273107e-04
> 41 KSP Residual norm 1.618672399310e-11
> 41 KSP preconditioned resid norm 1.618672399310e-11 true resid norm
> 2.034502467604e-07 ||r(i)||/||b|| 6.857299215548e-04
> 42 KSP Residual norm 1.378885267147e-11
> 42 KSP preconditioned resid norm 1.378885267147e-11 true resid norm
> 1.689714775892e-07 ||r(i)||/||b|| 5.695190834970e-04
> 43 KSP Residual norm 1.179601185051e-11
> 43 KSP preconditioned resid norm 1.179601185051e-11 true resid norm
> 1.483783124701e-07 ||r(i)||/||b|| 5.001097329234e-04
> 44 KSP Residual norm 9.928908801463e-12
> 44 KSP preconditioned resid norm 9.928908801463e-12 true resid norm
> 1.276885725440e-07 ||r(i)||/||b|| 4.303748765523e-04
> 45 KSP Residual norm 8.512555069524e-12
> 45 KSP preconditioned resid norm 8.512555069524e-12 true resid norm
> 1.020939171370e-07 ||r(i)||/||b|| 3.441079817022e-04
> 46 KSP Residual norm 7.600214409478e-12
> 46 KSP preconditioned resid norm 7.600214409478e-12 true resid norm
> 8.873369034852e-08 ||r(i)||/||b|| 2.990772805186e-04
> 47 KSP Residual norm 6.577382322417e-12
> 47 KSP preconditioned resid norm 6.577382322417e-12 true resid norm
> 8.285748008098e-08 ||r(i)||/||b|| 2.792714888326e-04
> 48 KSP Residual norm 5.701374824383e-12
> 48 KSP preconditioned resid norm 5.701374824383e-12 true resid norm
> 6.754179947516e-08 ||r(i)||/||b|| 2.276499222452e-04
> 49 KSP Residual norm 5.201909133133e-12
> 49 KSP preconditioned resid norm 5.201909133133e-12 true resid norm
> 5.807496359589e-08 ||r(i)||/||b|| 1.957419116714e-04
> 50 KSP Residual norm 4.803415857522e-12
> 50 KSP preconditioned resid norm 4.803415857522e-12 true resid norm
> 5.648916764539e-08 ||r(i)||/||b|| 1.903969796791e-04
> 51 KSP Residual norm 4.213404847970e-12
> 51 KSP preconditioned resid norm 4.213404847970e-12 true resid norm
> 5.312367187239e-08 ||r(i)||/||b|| 1.790535618699e-04
> 52 KSP Residual norm 3.583723287544e-12
> 52 KSP preconditioned resid norm 3.583723287544e-12 true resid norm
> 4.482581392980e-08 ||r(i)||/||b|| 1.510855964009e-04
> 53 KSP Residual norm 3.055433343791e-12
> 53 KSP preconditioned resid norm 3.055433343791e-12 true resid norm
> 3.733365588478e-08 ||r(i)||/||b|| 1.258332458617e-04
> 54 KSP Residual norm 2.632415195178e-12
> 54 KSP preconditioned resid norm 2.632415195178e-12 true resid norm
> 3.259461418510e-08 ||r(i)||/||b|| 1.098602856676e-04
> 55 KSP Residual norm 2.358953414259e-12
> 55 KSP preconditioned resid norm 2.358953414259e-12 true resid norm
> 2.755891784400e-08 ||r(i)||/||b|| 9.288744974367e-05
> 56 KSP Residual norm 2.131084837625e-12
> 56 KSP preconditioned resid norm 2.131084837625e-12 true resid norm
> 2.435335186857e-08 ||r(i)||/||b|| 8.208307599694e-05
> 57 KSP Residual norm 1.906113965754e-12
> 57 KSP preconditioned resid norm 1.906113965754e-12 true resid norm
> 2.234444331024e-08 ||r(i)||/||b|| 7.531204116139e-05
> 58 KSP Residual norm 1.666029207693e-12
> 58 KSP preconditioned resid norm 1.666029207693e-12 true resid norm
> 2.187608862971e-08 ||r(i)||/||b|| 7.373344971974e-05
> 59 KSP Residual norm 1.397594452516e-12
> 59 KSP preconditioned resid norm 1.397594452516e-12 true resid norm
> 1.884281030995e-08 ||r(i)||/||b|| 6.350977224879e-05
> 60 KSP Residual norm 1.000720808126e-12
> 60 KSP preconditioned resid norm 1.000720808126e-12 true resid norm
> 1.495264965711e-08 ||r(i)||/||b|| 5.039796923166e-05
> 61 KSP Residual norm 8.368281175845e-13
> 61 KSP preconditioned resid norm 8.368281175845e-13 true resid norm
> 1.198110007063e-08 ||r(i)||/||b|| 4.038234871863e-05
> 62 KSP Residual norm 6.975866756145e-13
> 62 KSP preconditioned resid norm 6.975866756145e-13 true resid norm
> 8.713139114636e-09 ||r(i)||/||b|| 2.936767242465e-05
> 63 KSP Residual norm 6.055685829863e-13
> 63 KSP preconditioned resid norm 6.055685829863e-13 true resid norm
> 7.208696201852e-09 ||r(i)||/||b|| 2.429694119186e-05
> 64 KSP Residual norm 5.321880026091e-13
> 64 KSP preconditioned resid norm 5.321880026091e-13 true resid norm
> 6.450668328122e-09 ||r(i)||/||b|| 2.174200502115e-05
> 65 KSP Residual norm 4.652520314729e-13
> 65 KSP preconditioned resid norm 4.652520314729e-13 true resid norm
> 5.767198491137e-09 ||r(i)||/||b|| 1.943836703022e-05
> 66 KSP Residual norm 4.095554125778e-13
> 66 KSP preconditioned resid norm 4.095554125778e-13 true resid norm
> 5.117612129931e-09 ||r(i)||/||b|| 1.724893343844e-05
> 67 KSP Residual norm 3.575261581129e-13
> 67 KSP preconditioned resid norm 3.575261581129e-13 true resid norm
> 4.479181564454e-09 ||r(i)||/||b|| 1.509710050359e-05
> 68 KSP Residual norm 3.109053579360e-13
> 68 KSP preconditioned resid norm 3.109053579360e-13 true resid norm
> 3.779777902208e-09 ||r(i)||/||b|| 1.273975748689e-05
> 69 KSP Residual norm 2.721884401030e-13
> 69 KSP preconditioned resid norm 2.721884401030e-13 true resid norm
> 3.393605957929e-09 ||r(i)||/||b|| 1.143816330711e-05
> Linear solve converged due to CONVERGED_RTOL iterations 69
> KSP Object: 1 MPI processes
>  type: gmres
>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>    GMRES: happy breakdown tolerance 1e-30
>  maximum iterations=10000, initial guess is zero
>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>  left preconditioning
>  using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>  type: ilu
>    ILU: out-of-place factorization
>    0 levels of fill
>    tolerance for zero pivot 2.22045e-14
>    matrix ordering: natural
>    factor fill ratio given 1, needed 1
>      Factored matrix follows:
>        Mat Object:         1 MPI processes
>          type: seqaij
>          rows=16384, cols=16384
>          package used to perform factorization: petsc
>          total: nonzeros=81920, allocated nonzeros=81920
>          total number of mallocs used during MatSetValues calls =0
>            not using I-node routines
>  linear system matrix = precond matrix:
>  Mat Object:   1 MPI processes
>    type: seqaij
>    rows=16384, cols=16384
>    total: nonzeros=81920, allocated nonzeros=81920
>    total number of mallocs used during MatSetValues calls =0
>      has attached null space
>      not using I-node routines
>
> 4)
>
>
>
> Linear solve converged due to CONVERGED_ITS iterations 1
> KSP Object: 1 MPI processes
>  type: preonly
>  maximum iterations=10000, initial guess is zero
>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>  left preconditioning
>  using NONE norm type for convergence test
> PC Object: 1 MPI processes
>  type: lu
>    LU: out-of-place factorization
>    tolerance for zero pivot 2.22045e-14
>    matrix ordering: nd
>    factor fill ratio given 5, needed 9.6582
>      Factored matrix follows:
>        Mat Object:         1 MPI processes
>          type: seqaij
>          rows=16384, cols=16384
>          package used to perform factorization: petsc
>          total: nonzeros=791200, allocated nonzeros=791200
>          total number of mallocs used during MatSetValues calls =0
>            not using I-node routines
>  linear system matrix = precond matrix:
>  Mat Object:   1 MPI processes
>    type: seqaij
>    rows=16384, cols=16384
>    total: nonzeros=81920, allocated nonzeros=81920
>    total number of mallocs used during MatSetValues calls =0
>      has attached null space
>      not using I-node routines
>
>
>
> On Tuesday 01 September 2015 03:40:14 Matthew Knepley wrote:
>
> > On Tue, Sep 1, 2015 at 2:22 AM, Filippo Leonardi <
>
> >
>
> > filippo.leonardi at sam.math.ethz.ch> wrote:
>
> > > Dear PETSc Users,
>
> > >
>
> > >
>
> > >
>
> > > I want to use multigrid to solve uniform (just Laplace) poisson in
> 2D/3D
>
> > > on cartesian, non-uniform meshes with a standard 5 (7)-points stencil
> FD.
>
> > >
>
> > >
>
> > >
>
> > > I always scaled my Poisson matrix like in the doc examples, i.e.
>
> > > multiplying by dx*dy (so that in ComputeRHS I need to scale b, in A*x
> = b,
>
> > > as well). This always worked properly with both MG/GAMG and with
> galerkin
>
> > > matrices.
>
> > >
>
> > >
>
> > >
>
> > > Now I'd like to use non-uniform meshes, therefore the scaling is
>
> > > non-uniform. However I cannot get my matrices to scale properly with
> any
>
> > > sort of multigrid.
>
> > >
>
> > >
>
> > >
>
> > > One would think that without scaling, i.e. solving the original
> system, at
>
> > > least MG+galerkin or GAMG should work anyways provided the matrix A
> and b
>
> > > are consistent.
>
> > >
>
> > >
>
> > >
>
> > > I tried without boundaries (i.e. torus), so this is not the problem.
>
> >
>
> > The point of that scaling is to make the boundary values the same size as
>
> > the residuals. If you have
>
> > no boundary values, the problem is scale invariant since its linear. It
>
> > would be nice to scale it so everything
>
> > is about size 1.
>
> >
>
> > When asking about convergence always run with -ksp_view
>
> > -ksp_monitor_true_residual -ksp_converged_reason
>
> >
>
> > Thanks,
>
> >
>
> > Matt
>
> >
>
> > > Anyone did/knows how to do this properly?
>
> > >
>
> > >
>
> > >
>
> > > Thanks,
>
> > >
>
> > > F
>
>
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/7856a540/attachment-0001.html>

From knepley at gmail.com  Tue Sep  1 05:13:54 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 Sep 2015 05:13:54 -0500
Subject: [petsc-users] 2D Poisson on nonuniform meshes
In-Reply-To: <9125684.etS2qxccK9@besikowitch-iii>
References: <3634751.Hfn5OcL7V3@besikowitch-iii>
	<5342199.ihN3W0jbJW@besikowitch-iii>
	<CAMYG4GnvdUhFhBEpsFrGb7vtH24us3TU-8LMuT4WWUT=z-6evw@mail.gmail.com>
	<9125684.etS2qxccK9@besikowitch-iii>
Message-ID: <CAMYG4GkBKPgOoEQhqGC_ct+jS3p036xCYjUsikryZYUjRQ0m0g@mail.gmail.com>

On Tue, Sep 1, 2015 at 4:38 AM, Filippo Leonardi <
filippo.leonardi at sam.math.ethz.ch> wrote:

> On Tuesday 01 September 2015 04:15:50 you wrote:
>
> > On Tue, Sep 1, 2015 at 4:12 AM, Filippo Leonardi <
>
> >
>
> > filippo.leonardi at sam.math.ethz.ch> wrote:
>
> > > Thanks Matt,
>
> > >
>
> > >
>
> > >
>
> > > I use the "natural scaling" of the Laplacian Lapl * x = b. The grid is
>
> > > *still* uniform.
>
> > >
>
> > >
>
> > >
>
> > > I give you a couple of results:
>
> > >
>
> > >
>
> > >
>
> > > 1 ) This is e.g. gmres + gamg + 4 levels. As you can see it converges
>
> > > after 2 iterations which is totally strange. The solution is incorrect.
>
> > >
>
> > >
>
> > >
>
> > > 2 ) This is mg + galerkin matrices + levels. Result is incorrect
>
> > >
>
> > >
>
> > >
>
> > > 3 ) Default: works fine
>
> > >
>
> > >
>
> > >
>
> > > 4) LU: works fine
>
> > >
>
> > >
>
> > >
>
> > > I remove the constant nullspace as usual.
>
> >
>
> > If you have a constant null space, how does 4) work? It should fail with
> a
>
> > zero pivot since
>
> > the matrix is singular.
>
>
>
> I do not care about the constant so any shift in the LU decomposition is
> fine and as far as I understood PETSc is robust in this respect.
>

1) The KSP view does not say it is shifting. Are you using the latest
release?

2) If it shifted, it would not solve in a single iterate.

3) Your GAMG results imply that something is wrong with the coarse solve.
This is exactly what would happen if
    that problem was not solved accurately (its off by > 10 orders of
magnitude).

It sounds like your operator is not singular, and its not the Laplacian
since it does not look like the Neumann version
has constants as a null space.

  Thanks,

     Matt


> >
>
> > Matt
>
> >
>
> > > I use the usual setcomputeoperator business and compute rhs.
>
> > >
>
> > >
>
> > >
>
> > > Instead, if I scale to Lapl * dx * dy * x = b * dx * dy (things work
> fine
>
> > > also with MG). This is fine with uniform grid but I do not know how to
> do
>
> > > that properly on nonuniform.
>
> > >
>
> > >
>
> > >
>
> > > Let me know if you need code pieces or other details.
>
> > >
>
> > >
>
> > >
>
> > > 1)
>
> > >
>
> > > 0 KSP Residual norm 1.793027353791e-05
>
> > > 0 KSP preconditioned resid norm 1.793027353791e-05 true resid norm
>
> > >
>
> > > 7.912994695029e-03 ||r(i)||/||b|| 1.000000000000e+00
>
> > >
>
> > > 1 KSP Residual norm 2.662212708666e-20
>
> > > 1 KSP preconditioned resid norm 2.662212708666e-20 true resid norm
>
> > >
>
> > > 3.120963188535e-02 ||r(i)||/||b|| 3.944098674167e+00
>
> > > Linear solve converged due to CONVERGED_RTOL iterations 1
>
> > > KSP Object: 1 MPI processes
>
> > >
>
> > > type: gmres
>
> > >
>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > >
>
> > > Orthogonalization with no iterative refinement
>
> > >
>
> > > GMRES: happy breakdown tolerance 1e-30
>
> > >
>
> > > maximum iterations=10000, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using PRECONDITIONED norm type for convergence test
>
> > >
>
> > > PC Object: 1 MPI processes
>
> > >
>
> > > type: gamg
>
> > >
>
> > > MG: type is MULTIPLICATIVE, levels=4 cycles=v
>
> > >
>
> > > Cycles per PCApply=1
>
> > > Using Galerkin computed coarse grid matrices
>
> > > GAMG specific options
>
> > >
>
> > > Threshold for dropping small values from graph 0
>
> > > AGG specific options
>
> > >
>
> > > Symmetric graph false
>
> > >
>
> > > Coarse grid solver -- level -------------------------------
>
> > >
>
> > > KSP Object: (mg_coarse_) 1 MPI processes
>
> > >
>
> > > type: gmres
>
> > >
>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > >
>
> > > Orthogonalization with no iterative refinement
>
> > >
>
> > > GMRES: happy breakdown tolerance 1e-30
>
> > >
>
> > > maximum iterations=1, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > PC Object: (mg_coarse_) 1 MPI processes
>
> > >
>
> > > type: bjacobi
>
> > >
>
> > > block Jacobi: number of blocks = 1
>
> > > Local solve is same for all blocks, in the following KSP and PC
>
> > >
>
> > > objects:
>
> > > KSP Object: (mg_coarse_sub_) 1 MPI processes
>
> > >
>
> > > type: preonly
>
> > > maximum iterations=1, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > PC Object: (mg_coarse_sub_) 1 MPI processes
>
> > >
>
> > > type: lu
>
> > >
>
> > > LU: out-of-place factorization
>
> > > tolerance for zero pivot 2.22045e-14
>
> > > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>
> > > matrix ordering: nd
>
> > > factor fill ratio given 5, needed 1.10953
>
> > >
>
> > > Factored matrix follows:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=33, cols=33
>
> > > package used to perform factorization: petsc
>
> > > total: nonzeros=1013, allocated nonzeros=1013
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > using I-node routines: found 21 nodes, limit used is 5
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=33, cols=33
>
> > > total: nonzeros=913, allocated nonzeros=913
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > not using I-node routines
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=33, cols=33
>
> > > total: nonzeros=913, allocated nonzeros=913
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > not using I-node routines
>
> > >
>
> > > Down solver (pre-smoother) on level 1 -------------------------------
>
> > >
>
> > > KSP Object: (mg_levels_1_) 1 MPI processes
>
> > >
>
> > > type: chebyshev
>
> > >
>
> > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > >
>
> > > 0.1; 0 1.1]
>
> > >
>
> > > KSP Object: (mg_levels_1_esteig_) 1 MPI processes
>
> > >
>
> > > type: gmres
>
> > >
>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > >
>
> > > Orthogonalization with no iterative refinement
>
> > >
>
> > > GMRES: happy breakdown tolerance 1e-30
>
> > >
>
> > > maximum iterations=10, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > maximum iterations=2
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using nonzero initial guess
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > PC Object: (mg_levels_1_) 1 MPI processes
>
> > >
>
> > > type: sor
>
> > >
>
> > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > >
>
> > > omega = 1
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=401, cols=401
>
> > > total: nonzeros=9777, allocated nonzeros=9777
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > not using I-node routines
>
> > >
>
> > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > Down solver (pre-smoother) on level 2 -------------------------------
>
> > >
>
> > > KSP Object: (mg_levels_2_) 1 MPI processes
>
> > >
>
> > > type: chebyshev
>
> > >
>
> > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > >
>
> > > 0.1; 0 1.1]
>
> > >
>
> > > KSP Object: (mg_levels_2_esteig_) 1 MPI processes
>
> > >
>
> > > type: gmres
>
> > >
>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > >
>
> > > Orthogonalization with no iterative refinement
>
> > >
>
> > > GMRES: happy breakdown tolerance 1e-30
>
> > >
>
> > > maximum iterations=10, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > maximum iterations=2
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using nonzero initial guess
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > PC Object: (mg_levels_2_) 1 MPI processes
>
> > >
>
> > > type: sor
>
> > >
>
> > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > >
>
> > > omega = 1
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=2296, cols=2296
>
> > > total: nonzeros=25038, allocated nonzeros=25038
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > not using I-node routines
>
> > >
>
> > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > Down solver (pre-smoother) on level 3 -------------------------------
>
> > >
>
> > > KSP Object: (mg_levels_3_) 1 MPI processes
>
> > >
>
> > > type: chebyshev
>
> > >
>
> > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > >
>
> > > 0.1; 0 1.1]
>
> > >
>
> > > KSP Object: (mg_levels_3_esteig_) 1 MPI processes
>
> > >
>
> > > type: gmres
>
> > >
>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > >
>
> > > Orthogonalization with no iterative refinement
>
> > >
>
> > > GMRES: happy breakdown tolerance 1e-30
>
> > >
>
> > > maximum iterations=10, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > maximum iterations=2
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using nonzero initial guess
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > PC Object: (mg_levels_3_) 1 MPI processes
>
> > >
>
> > > type: sor
>
> > >
>
> > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > >
>
> > > omega = 1
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=16384, cols=16384
>
> > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > has attached null space
>
> > > not using I-node routines
>
> > >
>
> > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=16384, cols=16384
>
> > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > has attached null space
>
> > > not using I-node routines
>
> > >
>
> > > 2)
>
> > >
>
> > > 0 KSP Residual norm 1.010294060491e-05
>
> > > 0 KSP preconditioned resid norm 1.010294060491e-05 true resid norm
>
> > >
>
> > > 1.086379257868e-03 ||r(i)||/||b|| 1.000000000000e+00
>
> > >
>
> > > 1 KSP Residual norm 8.803303876758e-20
>
> > > 1 KSP preconditioned resid norm 8.803303876758e-20 true resid norm
>
> > >
>
> > > 2.101751936637e-03 ||r(i)||/||b|| 1.934639235254e+00
>
> > > Linear solve converged due to CONVERGED_RTOL iterations 1
>
> > > KSP Object: 1 MPI processes
>
> > >
>
> > > type: gmres
>
> > >
>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > >
>
> > > Orthogonalization with no iterative refinement
>
> > >
>
> > > GMRES: happy breakdown tolerance 1e-30
>
> > >
>
> > > maximum iterations=10000, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using PRECONDITIONED norm type for convergence test
>
> > >
>
> > > PC Object: 1 MPI processes
>
> > >
>
> > > type: mg
>
> > >
>
> > > MG: type is MULTIPLICATIVE, levels=4 cycles=v
>
> > >
>
> > > Cycles per PCApply=1
>
> > > Using Galerkin computed coarse grid matrices
>
> > >
>
> > > Coarse grid solver -- level -------------------------------
>
> > >
>
> > > KSP Object: (mg_coarse_) 1 MPI processes
>
> > >
>
> > > type: preonly
>
> > > maximum iterations=1, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > PC Object: (mg_coarse_) 1 MPI processes
>
> > >
>
> > > type: lu
>
> > >
>
> > > LU: out-of-place factorization
>
> > > tolerance for zero pivot 2.22045e-14
>
> > > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>
> > > matrix ordering: nd
>
> > > factor fill ratio given 5, needed 4.83507
>
> > >
>
> > > Factored matrix follows:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=256, cols=256
>
> > > package used to perform factorization: petsc
>
> > > total: nonzeros=11140, allocated nonzeros=11140
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > not using I-node routines
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=256, cols=256
>
> > > total: nonzeros=2304, allocated nonzeros=2304
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > not using I-node routines
>
> > >
>
> > > Down solver (pre-smoother) on level 1 -------------------------------
>
> > >
>
> > > KSP Object: (mg_levels_1_) 1 MPI processes
>
> > >
>
> > > type: chebyshev
>
> > >
>
> > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > >
>
> > > 0.1; 0 1.1]
>
> > >
>
> > > KSP Object: (mg_levels_1_esteig_) 1 MPI processes
>
> > >
>
> > > type: gmres
>
> > >
>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > >
>
> > > Orthogonalization with no iterative refinement
>
> > >
>
> > > GMRES: happy breakdown tolerance 1e-30
>
> > >
>
> > > maximum iterations=10, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > maximum iterations=2
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using nonzero initial guess
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > PC Object: (mg_levels_1_) 1 MPI processes
>
> > >
>
> > > type: sor
>
> > >
>
> > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > >
>
> > > omega = 1
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=1024, cols=1024
>
> > > total: nonzeros=9216, allocated nonzeros=9216
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > not using I-node routines
>
> > >
>
> > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > Down solver (pre-smoother) on level 2 -------------------------------
>
> > >
>
> > > KSP Object: (mg_levels_2_) 1 MPI processes
>
> > >
>
> > > type: chebyshev
>
> > >
>
> > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > >
>
> > > 0.1; 0 1.1]
>
> > >
>
> > > KSP Object: (mg_levels_2_esteig_) 1 MPI processes
>
> > >
>
> > > type: gmres
>
> > >
>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > >
>
> > > Orthogonalization with no iterative refinement
>
> > >
>
> > > GMRES: happy breakdown tolerance 1e-30
>
> > >
>
> > > maximum iterations=10, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > maximum iterations=2
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using nonzero initial guess
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > PC Object: (mg_levels_2_) 1 MPI processes
>
> > >
>
> > > type: sor
>
> > >
>
> > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > >
>
> > > omega = 1
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=4096, cols=4096
>
> > > total: nonzeros=36864, allocated nonzeros=36864
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > not using I-node routines
>
> > >
>
> > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > Down solver (pre-smoother) on level 3 -------------------------------
>
> > >
>
> > > KSP Object: (mg_levels_3_) 1 MPI processes
>
> > >
>
> > > type: chebyshev
>
> > >
>
> > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > >
>
> > > 0.1; 0 1.1]
>
> > >
>
> > > KSP Object: (mg_levels_3_esteig_) 1 MPI processes
>
> > >
>
> > > type: gmres
>
> > >
>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > >
>
> > > Orthogonalization with no iterative refinement
>
> > >
>
> > > GMRES: happy breakdown tolerance 1e-30
>
> > >
>
> > > maximum iterations=10, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > maximum iterations=2
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using nonzero initial guess
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > PC Object: (mg_levels_3_) 1 MPI processes
>
> > >
>
> > > type: sor
>
> > >
>
> > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > >
>
> > > omega = 1
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=16384, cols=16384
>
> > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > has attached null space
>
> > > not using I-node routines
>
> > >
>
> > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=16384, cols=16384
>
> > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > has attached null space
>
> > > not using I-node routines
>
> > >
>
> > > 3)
>
> > >
>
> > >
>
> > >
>
> > > 0 KSP Residual norm 2.990548680394e-08
>
> > >
>
> > > 0 KSP preconditioned resid norm 2.990548680394e-08 true resid norm
>
> > >
>
> > > 2.966915112865e-04 ||r(i)||/||b|| 1.000000000000e+00
>
> > >
>
> > > 1 KSP Residual norm 2.368703753809e-08
>
> > > 1 KSP preconditioned resid norm 2.368703753809e-08 true resid norm
>
> > >
>
> > > 3.109667150576e-04 ||r(i)||/||b|| 1.048114634993e+00
>
> > >
>
> > > 2 KSP Residual norm 2.236515515253e-08
>
> > > 2 KSP preconditioned resid norm 2.236515515253e-08 true resid norm
>
> > >
>
> > > 2.655771471955e-04 ||r(i)||/||b|| 8.951289035670e-01
>
> > >
>
> > > 3 KSP Residual norm 1.957630940223e-08
>
> > > 3 KSP preconditioned resid norm 1.957630940223e-08 true resid norm
>
> > >
>
> > > 2.158250768646e-04 ||r(i)||/||b|| 7.274393390250e-01
>
> > >
>
> > > 4 KSP Residual norm 1.771416739221e-08
>
> > > 4 KSP preconditioned resid norm 1.771416739221e-08 true resid norm
>
> > >
>
> > > 2.075204223802e-04 ||r(i)||/||b|| 6.994484657830e-01
>
> > >
>
> > > 5 KSP Residual norm 1.617056152936e-08
>
> > > 5 KSP preconditioned resid norm 1.617056152936e-08 true resid norm
>
> > >
>
> > > 1.908090487285e-04 ||r(i)||/||b|| 6.431227098516e-01
>
> > >
>
> > > 6 KSP Residual norm 1.467170521607e-08
>
> > > 6 KSP preconditioned resid norm 1.467170521607e-08 true resid norm
>
> > >
>
> > > 1.683698119419e-04 ||r(i)||/||b|| 5.674911668750e-01
>
> > >
>
> > > 7 KSP Residual norm 1.321526248809e-08
>
> > > 7 KSP preconditioned resid norm 1.321526248809e-08 true resid norm
>
> > >
>
> > > 1.473617884651e-04 ||r(i)||/||b|| 4.966835344432e-01
>
> > >
>
> > > 8 KSP Residual norm 1.174828665876e-08
>
> > > 8 KSP preconditioned resid norm 1.174828665876e-08 true resid norm
>
> > >
>
> > > 1.458902362707e-04 ||r(i)||/||b|| 4.917236615167e-01
>
> > >
>
> > > 9 KSP Residual norm 1.012151625603e-08
>
> > > 9 KSP preconditioned resid norm 1.012151625603e-08 true resid norm
>
> > >
>
> > > 1.303866699420e-04 ||r(i)||/||b|| 4.394688253015e-01
>
> > > 10 KSP Residual norm 7.876439970777e-09
>
> > > 10 KSP preconditioned resid norm 7.876439970777e-09 true resid norm
>
> > > 1.162414266035e-04 ||r(i)||/||b|| 3.917922224990e-01
>
> > > 11 KSP Residual norm 4.981821461221e-09
>
> > > 11 KSP preconditioned resid norm 4.981821461221e-09 true resid norm
>
> > > 9.075981784793e-05 ||r(i)||/||b|| 3.059063518683e-01
>
> > > 12 KSP Residual norm 2.426217812448e-09
>
> > > 12 KSP preconditioned resid norm 2.426217812448e-09 true resid norm
>
> > > 5.145334699067e-05 ||r(i)||/||b|| 1.734237247556e-01
>
> > > 13 KSP Residual norm 1.049060114422e-09
>
> > > 13 KSP preconditioned resid norm 1.049060114422e-09 true resid norm
>
> > > 2.107527819651e-05 ||r(i)||/||b|| 7.103431475044e-02
>
> > > 14 KSP Residual norm 5.887222551677e-10
>
> > > 14 KSP preconditioned resid norm 5.887222551677e-10 true resid norm
>
> > > 9.047187295472e-06 ||r(i)||/||b|| 3.049358323817e-02
>
> > > 15 KSP Residual norm 4.536407095834e-10
>
> > > 15 KSP preconditioned resid norm 4.536407095834e-10 true resid norm
>
> > > 6.032647097712e-06 ||r(i)||/||b|| 2.033306268708e-02
>
> > > 16 KSP Residual norm 3.564018350025e-10
>
> > > 16 KSP preconditioned resid norm 3.564018350025e-10 true resid norm
>
> > > 5.116056274149e-06 ||r(i)||/||b|| 1.724368941991e-02
>
> > > 17 KSP Residual norm 2.744968239957e-10
>
> > > 17 KSP preconditioned resid norm 2.744968239957e-10 true resid norm
>
> > > 3.648248676660e-06 ||r(i)||/||b|| 1.229643767306e-02
>
> > > 18 KSP Residual norm 2.281847233955e-10
>
> > > 18 KSP preconditioned resid norm 2.281847233955e-10 true resid norm
>
> > > 2.900148270786e-06 ||r(i)||/||b|| 9.774962074955e-03
>
> > > 19 KSP Residual norm 1.881670741813e-10
>
> > > 19 KSP preconditioned resid norm 1.881670741813e-10 true resid norm
>
> > > 2.473850917169e-06 ||r(i)||/||b|| 8.338125032436e-03
>
> > > 20 KSP Residual norm 1.581922522755e-10
>
> > > 20 KSP preconditioned resid norm 1.581922522755e-10 true resid norm
>
> > > 1.965254213304e-06 ||r(i)||/||b|| 6.623897680059e-03
>
> > > 21 KSP Residual norm 1.349952347592e-10
>
> > > 21 KSP preconditioned resid norm 1.349952347592e-10 true resid norm
>
> > > 1.709953736235e-06 ||r(i)||/||b|| 5.763406336838e-03
>
> > > 22 KSP Residual norm 1.148754892680e-10
>
> > > 22 KSP preconditioned resid norm 1.148754892680e-10 true resid norm
>
> > > 1.452723184020e-06 ||r(i)||/||b|| 4.896409667135e-03
>
> > > 23 KSP Residual norm 9.889008860663e-11
>
> > > 23 KSP preconditioned resid norm 9.889008860663e-11 true resid norm
>
> > > 1.259317356950e-06 ||r(i)||/||b|| 4.244534504846e-03
>
> > > 24 KSP Residual norm 8.468070834423e-11
>
> > > 24 KSP preconditioned resid norm 8.468070834423e-11 true resid norm
>
> > > 1.097968496396e-06 ||r(i)||/||b|| 3.700707484468e-03
>
> > > 25 KSP Residual norm 7.340678871498e-11
>
> > > 25 KSP preconditioned resid norm 7.340678871498e-11 true resid norm
>
> > > 9.388293102378e-07 ||r(i)||/||b|| 3.164328181036e-03
>
> > > 26 KSP Residual norm 6.452497596580e-11
>
> > > 26 KSP preconditioned resid norm 6.452497596580e-11 true resid norm
>
> > > 7.907509627807e-07 ||r(i)||/||b|| 2.665229481464e-03
>
> > > 27 KSP Residual norm 5.817349095331e-11
>
> > > 27 KSP preconditioned resid norm 5.817349095331e-11 true resid norm
>
> > > 6.712549222809e-07 ||r(i)||/||b|| 2.262467568992e-03
>
> > > 28 KSP Residual norm 5.382715584394e-11
>
> > > 28 KSP preconditioned resid norm 5.382715584394e-11 true resid norm
>
> > > 5.883671521302e-07 ||r(i)||/||b|| 1.983093987351e-03
>
> > > 29 KSP Residual norm 5.072666199584e-11
>
> > > 29 KSP preconditioned resid norm 5.072666199584e-11 true resid norm
>
> > > 5.417512216577e-07 ||r(i)||/||b|| 1.825974795533e-03
>
> > > 30 KSP Residual norm 4.806650672230e-11
>
> > > 30 KSP preconditioned resid norm 4.806650672230e-11 true resid norm
>
> > > 5.151866485899e-07 ||r(i)||/||b|| 1.736438789084e-03
>
> > > 31 KSP Residual norm 4.675001898732e-11
>
> > > 31 KSP preconditioned resid norm 4.675001898732e-11 true resid norm
>
> > > 4.825947510614e-07 ||r(i)||/||b|| 1.626587659919e-03
>
> > > 32 KSP Residual norm 4.530536181592e-11
>
> > > 32 KSP preconditioned resid norm 4.530536181592e-11 true resid norm
>
> > > 4.648650267512e-07 ||r(i)||/||b|| 1.566829548764e-03
>
> > > 33 KSP Residual norm 4.357157235509e-11
>
> > > 33 KSP preconditioned resid norm 4.357157235509e-11 true resid norm
>
> > > 4.567209908707e-07 ||r(i)||/||b|| 1.539380041209e-03
>
> > > 34 KSP Residual norm 4.135715715343e-11
>
> > > 34 KSP preconditioned resid norm 4.135715715343e-11 true resid norm
>
> > > 4.469703503011e-07 ||r(i)||/||b|| 1.506515465721e-03
>
> > > 35 KSP Residual norm 3.851871986267e-11
>
> > > 35 KSP preconditioned resid norm 3.851871986267e-11 true resid norm
>
> > > 4.364097171314e-07 ||r(i)||/||b|| 1.470920806730e-03
>
> > > 36 KSP Residual norm 3.526035391634e-11
>
> > > 36 KSP preconditioned resid norm 3.526035391634e-11 true resid norm
>
> > > 4.065382561286e-07 ||r(i)||/||b|| 1.370238920439e-03
>
> > > 37 KSP Residual norm 3.144895012756e-11
>
> > > 37 KSP preconditioned resid norm 3.144895012756e-11 true resid norm
>
> > > 3.810168560499e-07 ||r(i)||/||b|| 1.284218932985e-03
>
> > > 38 KSP Residual norm 2.735239124106e-11
>
> > > 38 KSP preconditioned resid norm 2.735239124106e-11 true resid norm
>
> > > 3.407338934429e-07 ||r(i)||/||b|| 1.148445036278e-03
>
> > > 39 KSP Residual norm 2.306734554159e-11
>
> > > 39 KSP preconditioned resid norm 2.306734554159e-11 true resid norm
>
> > > 2.940484127641e-07 ||r(i)||/||b|| 9.910914251948e-04
>
> > > 40 KSP Residual norm 1.924842110922e-11
>
> > > 40 KSP preconditioned resid norm 1.924842110922e-11 true resid norm
>
> > > 2.443693779909e-07 ||r(i)||/||b|| 8.236480273107e-04
>
> > > 41 KSP Residual norm 1.618672399310e-11
>
> > > 41 KSP preconditioned resid norm 1.618672399310e-11 true resid norm
>
> > > 2.034502467604e-07 ||r(i)||/||b|| 6.857299215548e-04
>
> > > 42 KSP Residual norm 1.378885267147e-11
>
> > > 42 KSP preconditioned resid norm 1.378885267147e-11 true resid norm
>
> > > 1.689714775892e-07 ||r(i)||/||b|| 5.695190834970e-04
>
> > > 43 KSP Residual norm 1.179601185051e-11
>
> > > 43 KSP preconditioned resid norm 1.179601185051e-11 true resid norm
>
> > > 1.483783124701e-07 ||r(i)||/||b|| 5.001097329234e-04
>
> > > 44 KSP Residual norm 9.928908801463e-12
>
> > > 44 KSP preconditioned resid norm 9.928908801463e-12 true resid norm
>
> > > 1.276885725440e-07 ||r(i)||/||b|| 4.303748765523e-04
>
> > > 45 KSP Residual norm 8.512555069524e-12
>
> > > 45 KSP preconditioned resid norm 8.512555069524e-12 true resid norm
>
> > > 1.020939171370e-07 ||r(i)||/||b|| 3.441079817022e-04
>
> > > 46 KSP Residual norm 7.600214409478e-12
>
> > > 46 KSP preconditioned resid norm 7.600214409478e-12 true resid norm
>
> > > 8.873369034852e-08 ||r(i)||/||b|| 2.990772805186e-04
>
> > > 47 KSP Residual norm 6.577382322417e-12
>
> > > 47 KSP preconditioned resid norm 6.577382322417e-12 true resid norm
>
> > > 8.285748008098e-08 ||r(i)||/||b|| 2.792714888326e-04
>
> > > 48 KSP Residual norm 5.701374824383e-12
>
> > > 48 KSP preconditioned resid norm 5.701374824383e-12 true resid norm
>
> > > 6.754179947516e-08 ||r(i)||/||b|| 2.276499222452e-04
>
> > > 49 KSP Residual norm 5.201909133133e-12
>
> > > 49 KSP preconditioned resid norm 5.201909133133e-12 true resid norm
>
> > > 5.807496359589e-08 ||r(i)||/||b|| 1.957419116714e-04
>
> > > 50 KSP Residual norm 4.803415857522e-12
>
> > > 50 KSP preconditioned resid norm 4.803415857522e-12 true resid norm
>
> > > 5.648916764539e-08 ||r(i)||/||b|| 1.903969796791e-04
>
> > > 51 KSP Residual norm 4.213404847970e-12
>
> > > 51 KSP preconditioned resid norm 4.213404847970e-12 true resid norm
>
> > > 5.312367187239e-08 ||r(i)||/||b|| 1.790535618699e-04
>
> > > 52 KSP Residual norm 3.583723287544e-12
>
> > > 52 KSP preconditioned resid norm 3.583723287544e-12 true resid norm
>
> > > 4.482581392980e-08 ||r(i)||/||b|| 1.510855964009e-04
>
> > > 53 KSP Residual norm 3.055433343791e-12
>
> > > 53 KSP preconditioned resid norm 3.055433343791e-12 true resid norm
>
> > > 3.733365588478e-08 ||r(i)||/||b|| 1.258332458617e-04
>
> > > 54 KSP Residual norm 2.632415195178e-12
>
> > > 54 KSP preconditioned resid norm 2.632415195178e-12 true resid norm
>
> > > 3.259461418510e-08 ||r(i)||/||b|| 1.098602856676e-04
>
> > > 55 KSP Residual norm 2.358953414259e-12
>
> > > 55 KSP preconditioned resid norm 2.358953414259e-12 true resid norm
>
> > > 2.755891784400e-08 ||r(i)||/||b|| 9.288744974367e-05
>
> > > 56 KSP Residual norm 2.131084837625e-12
>
> > > 56 KSP preconditioned resid norm 2.131084837625e-12 true resid norm
>
> > > 2.435335186857e-08 ||r(i)||/||b|| 8.208307599694e-05
>
> > > 57 KSP Residual norm 1.906113965754e-12
>
> > > 57 KSP preconditioned resid norm 1.906113965754e-12 true resid norm
>
> > > 2.234444331024e-08 ||r(i)||/||b|| 7.531204116139e-05
>
> > > 58 KSP Residual norm 1.666029207693e-12
>
> > > 58 KSP preconditioned resid norm 1.666029207693e-12 true resid norm
>
> > > 2.187608862971e-08 ||r(i)||/||b|| 7.373344971974e-05
>
> > > 59 KSP Residual norm 1.397594452516e-12
>
> > > 59 KSP preconditioned resid norm 1.397594452516e-12 true resid norm
>
> > > 1.884281030995e-08 ||r(i)||/||b|| 6.350977224879e-05
>
> > > 60 KSP Residual norm 1.000720808126e-12
>
> > > 60 KSP preconditioned resid norm 1.000720808126e-12 true resid norm
>
> > > 1.495264965711e-08 ||r(i)||/||b|| 5.039796923166e-05
>
> > > 61 KSP Residual norm 8.368281175845e-13
>
> > > 61 KSP preconditioned resid norm 8.368281175845e-13 true resid norm
>
> > > 1.198110007063e-08 ||r(i)||/||b|| 4.038234871863e-05
>
> > > 62 KSP Residual norm 6.975866756145e-13
>
> > > 62 KSP preconditioned resid norm 6.975866756145e-13 true resid norm
>
> > > 8.713139114636e-09 ||r(i)||/||b|| 2.936767242465e-05
>
> > > 63 KSP Residual norm 6.055685829863e-13
>
> > > 63 KSP preconditioned resid norm 6.055685829863e-13 true resid norm
>
> > > 7.208696201852e-09 ||r(i)||/||b|| 2.429694119186e-05
>
> > > 64 KSP Residual norm 5.321880026091e-13
>
> > > 64 KSP preconditioned resid norm 5.321880026091e-13 true resid norm
>
> > > 6.450668328122e-09 ||r(i)||/||b|| 2.174200502115e-05
>
> > > 65 KSP Residual norm 4.652520314729e-13
>
> > > 65 KSP preconditioned resid norm 4.652520314729e-13 true resid norm
>
> > > 5.767198491137e-09 ||r(i)||/||b|| 1.943836703022e-05
>
> > > 66 KSP Residual norm 4.095554125778e-13
>
> > > 66 KSP preconditioned resid norm 4.095554125778e-13 true resid norm
>
> > > 5.117612129931e-09 ||r(i)||/||b|| 1.724893343844e-05
>
> > > 67 KSP Residual norm 3.575261581129e-13
>
> > > 67 KSP preconditioned resid norm 3.575261581129e-13 true resid norm
>
> > > 4.479181564454e-09 ||r(i)||/||b|| 1.509710050359e-05
>
> > > 68 KSP Residual norm 3.109053579360e-13
>
> > > 68 KSP preconditioned resid norm 3.109053579360e-13 true resid norm
>
> > > 3.779777902208e-09 ||r(i)||/||b|| 1.273975748689e-05
>
> > > 69 KSP Residual norm 2.721884401030e-13
>
> > > 69 KSP preconditioned resid norm 2.721884401030e-13 true resid norm
>
> > > 3.393605957929e-09 ||r(i)||/||b|| 1.143816330711e-05
>
> > > Linear solve converged due to CONVERGED_RTOL iterations 69
>
> > > KSP Object: 1 MPI processes
>
> > >
>
> > > type: gmres
>
> > >
>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > >
>
> > > Orthogonalization with no iterative refinement
>
> > >
>
> > > GMRES: happy breakdown tolerance 1e-30
>
> > >
>
> > > maximum iterations=10000, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using PRECONDITIONED norm type for convergence test
>
> > >
>
> > > PC Object: 1 MPI processes
>
> > >
>
> > > type: ilu
>
> > >
>
> > > ILU: out-of-place factorization
>
> > > 0 levels of fill
>
> > > tolerance for zero pivot 2.22045e-14
>
> > > matrix ordering: natural
>
> > > factor fill ratio given 1, needed 1
>
> > >
>
> > > Factored matrix follows:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=16384, cols=16384
>
> > > package used to perform factorization: petsc
>
> > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > not using I-node routines
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=16384, cols=16384
>
> > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > has attached null space
>
> > > not using I-node routines
>
> > >
>
> > > 4)
>
> > >
>
> > >
>
> > >
>
> > > Linear solve converged due to CONVERGED_ITS iterations 1
>
> > > KSP Object: 1 MPI processes
>
> > >
>
> > > type: preonly
>
> > > maximum iterations=10000, initial guess is zero
>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > left preconditioning
>
> > > using NONE norm type for convergence test
>
> > >
>
> > > PC Object: 1 MPI processes
>
> > >
>
> > > type: lu
>
> > >
>
> > > LU: out-of-place factorization
>
> > > tolerance for zero pivot 2.22045e-14
>
> > > matrix ordering: nd
>
> > > factor fill ratio given 5, needed 9.6582
>
> > >
>
> > > Factored matrix follows:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=16384, cols=16384
>
> > > package used to perform factorization: petsc
>
> > > total: nonzeros=791200, allocated nonzeros=791200
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > not using I-node routines
>
> > >
>
> > > linear system matrix = precond matrix:
>
> > > Mat Object: 1 MPI processes
>
> > >
>
> > > type: seqaij
>
> > > rows=16384, cols=16384
>
> > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > total number of mallocs used during MatSetValues calls =0
>
> > >
>
> > > has attached null space
>
> > > not using I-node routines
>
> > >
>
> > > On Tuesday 01 September 2015 03:40:14 Matthew Knepley wrote:
>
> > > > On Tue, Sep 1, 2015 at 2:22 AM, Filippo Leonardi <
>
> > > >
>
> > > > filippo.leonardi at sam.math.ethz.ch> wrote:
>
> > > > > Dear PETSc Users,
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > I want to use multigrid to solve uniform (just Laplace) poisson in
>
> > >
>
> > > 2D/3D
>
> > >
>
> > > > > on cartesian, non-uniform meshes with a standard 5 (7)-points
> stencil
>
> > >
>
> > > FD.
>
> > >
>
> > > > > I always scaled my Poisson matrix like in the doc examples, i.e.
>
> > > > >
>
> > > > > multiplying by dx*dy (so that in ComputeRHS I need to scale b, in
> A*x
>
> > >
>
> > > = b,
>
> > >
>
> > > > > as well). This always worked properly with both MG/GAMG and with
>
> > >
>
> > > galerkin
>
> > >
>
> > > > > matrices.
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Now I'd like to use non-uniform meshes, therefore the scaling is
>
> > > > >
>
> > > > > non-uniform. However I cannot get my matrices to scale properly
> with
>
> > >
>
> > > any
>
> > >
>
> > > > > sort of multigrid.
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > One would think that without scaling, i.e. solving the original
>
> > >
>
> > > system, at
>
> > >
>
> > > > > least MG+galerkin or GAMG should work anyways provided the matrix A
>
> > >
>
> > > and b
>
> > >
>
> > > > > are consistent.
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > I tried without boundaries (i.e. torus), so this is not the
> problem.
>
> > > >
>
> > > > The point of that scaling is to make the boundary values the same
> size
>
> > > > as
>
> > > >
>
> > > > the residuals. If you have
>
> > > >
>
> > > > no boundary values, the problem is scale invariant since its linear.
> It
>
> > > >
>
> > > > would be nice to scale it so everything
>
> > > >
>
> > > > is about size 1.
>
> > > >
>
> > > >
>
> > > >
>
> > > > When asking about convergence always run with -ksp_view
>
> > > >
>
> > > > -ksp_monitor_true_residual -ksp_converged_reason
>
> > > >
>
> > > >
>
> > > >
>
> > > > Thanks,
>
> > > >
>
> > > >
>
> > > >
>
> > > > Matt
>
> > > >
>
> > > > > Anyone did/knows how to do this properly?
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Thanks,
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > F
>
>
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/88d26b6f/attachment-0001.html>

From knepley at gmail.com  Tue Sep  1 06:32:38 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 Sep 2015 06:32:38 -0500
Subject: [petsc-users] 2D Poisson on nonuniform meshes
In-Reply-To: <2620235.Qtx0xDWqJW@besikowitch-iii>
References: <3634751.Hfn5OcL7V3@besikowitch-iii>
	<9125684.etS2qxccK9@besikowitch-iii>
	<CAMYG4GkBKPgOoEQhqGC_ct+jS3p036xCYjUsikryZYUjRQ0m0g@mail.gmail.com>
	<2620235.Qtx0xDWqJW@besikowitch-iii>
Message-ID: <CAMYG4GncTzqDzMe3s86e6a0A7RsZhGT7F-p_sqjgB_mOT-cL0Q@mail.gmail.com>

On Tue, Sep 1, 2015 at 6:24 AM, Filippo Leonardi <
filippo.leonardi at sam.math.ethz.ch> wrote:

> On Tuesday 01 September 2015 05:13:54 you wrote:
>
> > On Tue, Sep 1, 2015 at 4:38 AM, Filippo Leonardi <
>
> >
>
> > filippo.leonardi at sam.math.ethz.ch> wrote:
>
> > > On Tuesday 01 September 2015 04:15:50 you wrote:
>
> > > > On Tue, Sep 1, 2015 at 4:12 AM, Filippo Leonardi <
>
> > > >
>
> > > > filippo.leonardi at sam.math.ethz.ch> wrote:
>
> > > > > Thanks Matt,
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > I use the "natural scaling" of the Laplacian Lapl * x = b. The
> grid is
>
> > > > >
>
> > > > > *still* uniform.
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > I give you a couple of results:
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1 ) This is e.g. gmres + gamg + 4 levels. As you can see it
> converges
>
> > > > >
>
> > > > > after 2 iterations which is totally strange. The solution is
>
> > > > > incorrect.
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 2 ) This is mg + galerkin matrices + levels. Result is incorrect
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 3 ) Default: works fine
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 4) LU: works fine
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > I remove the constant nullspace as usual.
>
> > > >
>
> > > > If you have a constant null space, how does 4) work? It should fail
> with
>
> > >
>
> > > a
>
> > >
>
> > > > zero pivot since
>
> > > >
>
> > > > the matrix is singular.
>
> > >
>
> > > I do not care about the constant so any shift in the LU decomposition
> is
>
> > > fine and as far as I understood PETSc is robust in this respect.
>
> >
>
> > 1) The KSP view does not say it is shifting. Are you using the latest
>
> > release?
>
>
>
> yes, 3.6. Does PETSc warn for that even if I set the nullspace? I can also
> check MUMPS or something else.
>

I am not sure what you think PETSc does here. If shifting were enabled, it
would add some
diagonal matrix to the input matrix and continue the factorization. This
would mean that the
factors were not, in fact, a factorization of the input matrix, and you
would not get the exact
solution in one iterate.


> >
>
> > 2) If it shifted, it would not solve in a single iterate.
>
>
>
> even with preonly?
>

You would have a large residual. Do you?


> >
>
> > 3) Your GAMG results imply that something is wrong with the coarse solve.
>
> > This is exactly what would happen if
>
> > that problem was not solved accurately (its off by > 10 orders of
>
> > magnitude).
>
>
>
> yes, but GAMG builds is own coarse solvers so either the problem is
> already in the definition of A and b (likely) or it is a bug in gamg.
>

Yes. GAMG uses the constants to build the basis, on the assumption that
they are in the (near) nullspace of the
operator with no boundary conditions. Since this is far off, I think this
must not be true for your A.


> >
>
> > It sounds like your operator is not singular, and its not the Laplacian
>
> > since it does not look like the Neumann version
>
> > has constants as a null space.
>
>
>
> I'm using periodic boundaries, and constants are in kern(A)
>

Did you check?

   Matt


> Thanks a lot for your time.
>
> >
>
> > Thanks,
>
> >
>
> > Matt
>
> >
>
> > > > Matt
>
> > > >
>
> > > > > I use the usual setcomputeoperator business and compute rhs.
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Instead, if I scale to Lapl * dx * dy * x = b * dx * dy (things
> work
>
> > >
>
> > > fine
>
> > >
>
> > > > > also with MG). This is fine with uniform grid but I do not know
> how to
>
> > >
>
> > > do
>
> > >
>
> > > > > that properly on nonuniform.
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Let me know if you need code pieces or other details.
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1)
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 0 KSP Residual norm 1.793027353791e-05
>
> > > > >
>
> > > > > 0 KSP preconditioned resid norm 1.793027353791e-05 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 7.912994695029e-03 ||r(i)||/||b|| 1.000000000000e+00
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1 KSP Residual norm 2.662212708666e-20
>
> > > > >
>
> > > > > 1 KSP preconditioned resid norm 2.662212708666e-20 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 3.120963188535e-02 ||r(i)||/||b|| 3.944098674167e+00
>
> > > > >
>
> > > > > Linear solve converged due to CONVERGED_RTOL iterations 1
>
> > > > >
>
> > > > > KSP Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gmres
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Orthogonalization with no iterative refinement
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: happy breakdown tolerance 1e-30
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=10000, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using PRECONDITIONED norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gamg
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > MG: type is MULTIPLICATIVE, levels=4 cycles=v
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Cycles per PCApply=1
>
> > > > >
>
> > > > > Using Galerkin computed coarse grid matrices
>
> > > > >
>
> > > > > GAMG specific options
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Threshold for dropping small values from graph 0
>
> > > > >
>
> > > > > AGG specific options
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Symmetric graph false
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Coarse grid solver -- level -------------------------------
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_coarse_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gmres
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Orthogonalization with no iterative refinement
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: happy breakdown tolerance 1e-30
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=1, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: (mg_coarse_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: bjacobi
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > block Jacobi: number of blocks = 1
>
> > > > >
>
> > > > > Local solve is same for all blocks, in the following KSP and PC
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > objects:
>
> > > > >
>
> > > > > KSP Object: (mg_coarse_sub_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: preonly
>
> > > > >
>
> > > > > maximum iterations=1, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: (mg_coarse_sub_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: lu
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > LU: out-of-place factorization
>
> > > > >
>
> > > > > tolerance for zero pivot 2.22045e-14
>
> > > > >
>
> > > > > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>
> > > > >
>
> > > > > matrix ordering: nd
>
> > > > >
>
> > > > > factor fill ratio given 5, needed 1.10953
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Factored matrix follows:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=33, cols=33
>
> > > > >
>
> > > > > package used to perform factorization: petsc
>
> > > > >
>
> > > > > total: nonzeros=1013, allocated nonzeros=1013
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > using I-node routines: found 21 nodes, limit used is 5
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=33, cols=33
>
> > > > >
>
> > > > > total: nonzeros=913, allocated nonzeros=913
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=33, cols=33
>
> > > > >
>
> > > > > total: nonzeros=913, allocated nonzeros=913
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Down solver (pre-smoother) on level 1
> -------------------------------
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_1_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: chebyshev
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > > >
>
> > > > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 0.1; 0 1.1]
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_1_esteig_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gmres
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Orthogonalization with no iterative refinement
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: happy breakdown tolerance 1e-30
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=10, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=2
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using nonzero initial guess
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: (mg_levels_1_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: sor
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > omega = 1
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=401, cols=401
>
> > > > >
>
> > > > > total: nonzeros=9777, allocated nonzeros=9777
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > > >
>
> > > > > Down solver (pre-smoother) on level 2
> -------------------------------
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_2_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: chebyshev
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > > >
>
> > > > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 0.1; 0 1.1]
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_2_esteig_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gmres
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Orthogonalization with no iterative refinement
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: happy breakdown tolerance 1e-30
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=10, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=2
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using nonzero initial guess
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: (mg_levels_2_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: sor
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > omega = 1
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=2296, cols=2296
>
> > > > >
>
> > > > > total: nonzeros=25038, allocated nonzeros=25038
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > > >
>
> > > > > Down solver (pre-smoother) on level 3
> -------------------------------
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_3_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: chebyshev
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > > >
>
> > > > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 0.1; 0 1.1]
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_3_esteig_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gmres
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Orthogonalization with no iterative refinement
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: happy breakdown tolerance 1e-30
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=10, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=2
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using nonzero initial guess
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: (mg_levels_3_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: sor
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > omega = 1
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=16384, cols=16384
>
> > > > >
>
> > > > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > has attached null space
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=16384, cols=16384
>
> > > > >
>
> > > > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > has attached null space
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 2)
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 0 KSP Residual norm 1.010294060491e-05
>
> > > > >
>
> > > > > 0 KSP preconditioned resid norm 1.010294060491e-05 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1.086379257868e-03 ||r(i)||/||b|| 1.000000000000e+00
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1 KSP Residual norm 8.803303876758e-20
>
> > > > >
>
> > > > > 1 KSP preconditioned resid norm 8.803303876758e-20 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 2.101751936637e-03 ||r(i)||/||b|| 1.934639235254e+00
>
> > > > >
>
> > > > > Linear solve converged due to CONVERGED_RTOL iterations 1
>
> > > > >
>
> > > > > KSP Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gmres
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Orthogonalization with no iterative refinement
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: happy breakdown tolerance 1e-30
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=10000, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using PRECONDITIONED norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: mg
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > MG: type is MULTIPLICATIVE, levels=4 cycles=v
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Cycles per PCApply=1
>
> > > > >
>
> > > > > Using Galerkin computed coarse grid matrices
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Coarse grid solver -- level -------------------------------
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_coarse_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: preonly
>
> > > > >
>
> > > > > maximum iterations=1, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: (mg_coarse_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: lu
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > LU: out-of-place factorization
>
> > > > >
>
> > > > > tolerance for zero pivot 2.22045e-14
>
> > > > >
>
> > > > > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>
> > > > >
>
> > > > > matrix ordering: nd
>
> > > > >
>
> > > > > factor fill ratio given 5, needed 4.83507
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Factored matrix follows:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=256, cols=256
>
> > > > >
>
> > > > > package used to perform factorization: petsc
>
> > > > >
>
> > > > > total: nonzeros=11140, allocated nonzeros=11140
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=256, cols=256
>
> > > > >
>
> > > > > total: nonzeros=2304, allocated nonzeros=2304
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Down solver (pre-smoother) on level 1
> -------------------------------
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_1_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: chebyshev
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > > >
>
> > > > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 0.1; 0 1.1]
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_1_esteig_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gmres
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Orthogonalization with no iterative refinement
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: happy breakdown tolerance 1e-30
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=10, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=2
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using nonzero initial guess
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: (mg_levels_1_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: sor
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > omega = 1
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=1024, cols=1024
>
> > > > >
>
> > > > > total: nonzeros=9216, allocated nonzeros=9216
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > > >
>
> > > > > Down solver (pre-smoother) on level 2
> -------------------------------
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_2_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: chebyshev
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > > >
>
> > > > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 0.1; 0 1.1]
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_2_esteig_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gmres
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Orthogonalization with no iterative refinement
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: happy breakdown tolerance 1e-30
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=10, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=2
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using nonzero initial guess
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: (mg_levels_2_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: sor
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > omega = 1
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=4096, cols=4096
>
> > > > >
>
> > > > > total: nonzeros=36864, allocated nonzeros=36864
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > > >
>
> > > > > Down solver (pre-smoother) on level 3
> -------------------------------
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_3_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: chebyshev
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Chebyshev: eigenvalue estimates: min = -1.79769e+307, max = -inf
>
> > > > >
>
> > > > > Chebyshev: eigenvalues estimated using gmres with translations [0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 0.1; 0 1.1]
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > KSP Object: (mg_levels_3_esteig_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gmres
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Orthogonalization with no iterative refinement
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: happy breakdown tolerance 1e-30
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=10, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=2
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using nonzero initial guess
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: (mg_levels_3_) 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: sor
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > SOR: type = local_symmetric, iterations = 1, local iterations = 1,
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > omega = 1
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=16384, cols=16384
>
> > > > >
>
> > > > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > has attached null space
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Up solver (post-smoother) same as down solver (pre-smoother)
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=16384, cols=16384
>
> > > > >
>
> > > > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > has attached null space
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 3)
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 0 KSP Residual norm 2.990548680394e-08
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 0 KSP preconditioned resid norm 2.990548680394e-08 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 2.966915112865e-04 ||r(i)||/||b|| 1.000000000000e+00
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1 KSP Residual norm 2.368703753809e-08
>
> > > > >
>
> > > > > 1 KSP preconditioned resid norm 2.368703753809e-08 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 3.109667150576e-04 ||r(i)||/||b|| 1.048114634993e+00
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 2 KSP Residual norm 2.236515515253e-08
>
> > > > >
>
> > > > > 2 KSP preconditioned resid norm 2.236515515253e-08 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 2.655771471955e-04 ||r(i)||/||b|| 8.951289035670e-01
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 3 KSP Residual norm 1.957630940223e-08
>
> > > > >
>
> > > > > 3 KSP preconditioned resid norm 1.957630940223e-08 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 2.158250768646e-04 ||r(i)||/||b|| 7.274393390250e-01
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 4 KSP Residual norm 1.771416739221e-08
>
> > > > >
>
> > > > > 4 KSP preconditioned resid norm 1.771416739221e-08 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 2.075204223802e-04 ||r(i)||/||b|| 6.994484657830e-01
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 5 KSP Residual norm 1.617056152936e-08
>
> > > > >
>
> > > > > 5 KSP preconditioned resid norm 1.617056152936e-08 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1.908090487285e-04 ||r(i)||/||b|| 6.431227098516e-01
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 6 KSP Residual norm 1.467170521607e-08
>
> > > > >
>
> > > > > 6 KSP preconditioned resid norm 1.467170521607e-08 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1.683698119419e-04 ||r(i)||/||b|| 5.674911668750e-01
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 7 KSP Residual norm 1.321526248809e-08
>
> > > > >
>
> > > > > 7 KSP preconditioned resid norm 1.321526248809e-08 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1.473617884651e-04 ||r(i)||/||b|| 4.966835344432e-01
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 8 KSP Residual norm 1.174828665876e-08
>
> > > > >
>
> > > > > 8 KSP preconditioned resid norm 1.174828665876e-08 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1.458902362707e-04 ||r(i)||/||b|| 4.917236615167e-01
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 9 KSP Residual norm 1.012151625603e-08
>
> > > > >
>
> > > > > 9 KSP preconditioned resid norm 1.012151625603e-08 true resid norm
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 1.303866699420e-04 ||r(i)||/||b|| 4.394688253015e-01
>
> > > > >
>
> > > > > 10 KSP Residual norm 7.876439970777e-09
>
> > > > >
>
> > > > > 10 KSP preconditioned resid norm 7.876439970777e-09 true resid norm
>
> > > > >
>
> > > > > 1.162414266035e-04 ||r(i)||/||b|| 3.917922224990e-01
>
> > > > >
>
> > > > > 11 KSP Residual norm 4.981821461221e-09
>
> > > > >
>
> > > > > 11 KSP preconditioned resid norm 4.981821461221e-09 true resid norm
>
> > > > >
>
> > > > > 9.075981784793e-05 ||r(i)||/||b|| 3.059063518683e-01
>
> > > > >
>
> > > > > 12 KSP Residual norm 2.426217812448e-09
>
> > > > >
>
> > > > > 12 KSP preconditioned resid norm 2.426217812448e-09 true resid norm
>
> > > > >
>
> > > > > 5.145334699067e-05 ||r(i)||/||b|| 1.734237247556e-01
>
> > > > >
>
> > > > > 13 KSP Residual norm 1.049060114422e-09
>
> > > > >
>
> > > > > 13 KSP preconditioned resid norm 1.049060114422e-09 true resid norm
>
> > > > >
>
> > > > > 2.107527819651e-05 ||r(i)||/||b|| 7.103431475044e-02
>
> > > > >
>
> > > > > 14 KSP Residual norm 5.887222551677e-10
>
> > > > >
>
> > > > > 14 KSP preconditioned resid norm 5.887222551677e-10 true resid norm
>
> > > > >
>
> > > > > 9.047187295472e-06 ||r(i)||/||b|| 3.049358323817e-02
>
> > > > >
>
> > > > > 15 KSP Residual norm 4.536407095834e-10
>
> > > > >
>
> > > > > 15 KSP preconditioned resid norm 4.536407095834e-10 true resid norm
>
> > > > >
>
> > > > > 6.032647097712e-06 ||r(i)||/||b|| 2.033306268708e-02
>
> > > > >
>
> > > > > 16 KSP Residual norm 3.564018350025e-10
>
> > > > >
>
> > > > > 16 KSP preconditioned resid norm 3.564018350025e-10 true resid norm
>
> > > > >
>
> > > > > 5.116056274149e-06 ||r(i)||/||b|| 1.724368941991e-02
>
> > > > >
>
> > > > > 17 KSP Residual norm 2.744968239957e-10
>
> > > > >
>
> > > > > 17 KSP preconditioned resid norm 2.744968239957e-10 true resid norm
>
> > > > >
>
> > > > > 3.648248676660e-06 ||r(i)||/||b|| 1.229643767306e-02
>
> > > > >
>
> > > > > 18 KSP Residual norm 2.281847233955e-10
>
> > > > >
>
> > > > > 18 KSP preconditioned resid norm 2.281847233955e-10 true resid norm
>
> > > > >
>
> > > > > 2.900148270786e-06 ||r(i)||/||b|| 9.774962074955e-03
>
> > > > >
>
> > > > > 19 KSP Residual norm 1.881670741813e-10
>
> > > > >
>
> > > > > 19 KSP preconditioned resid norm 1.881670741813e-10 true resid norm
>
> > > > >
>
> > > > > 2.473850917169e-06 ||r(i)||/||b|| 8.338125032436e-03
>
> > > > >
>
> > > > > 20 KSP Residual norm 1.581922522755e-10
>
> > > > >
>
> > > > > 20 KSP preconditioned resid norm 1.581922522755e-10 true resid norm
>
> > > > >
>
> > > > > 1.965254213304e-06 ||r(i)||/||b|| 6.623897680059e-03
>
> > > > >
>
> > > > > 21 KSP Residual norm 1.349952347592e-10
>
> > > > >
>
> > > > > 21 KSP preconditioned resid norm 1.349952347592e-10 true resid norm
>
> > > > >
>
> > > > > 1.709953736235e-06 ||r(i)||/||b|| 5.763406336838e-03
>
> > > > >
>
> > > > > 22 KSP Residual norm 1.148754892680e-10
>
> > > > >
>
> > > > > 22 KSP preconditioned resid norm 1.148754892680e-10 true resid norm
>
> > > > >
>
> > > > > 1.452723184020e-06 ||r(i)||/||b|| 4.896409667135e-03
>
> > > > >
>
> > > > > 23 KSP Residual norm 9.889008860663e-11
>
> > > > >
>
> > > > > 23 KSP preconditioned resid norm 9.889008860663e-11 true resid norm
>
> > > > >
>
> > > > > 1.259317356950e-06 ||r(i)||/||b|| 4.244534504846e-03
>
> > > > >
>
> > > > > 24 KSP Residual norm 8.468070834423e-11
>
> > > > >
>
> > > > > 24 KSP preconditioned resid norm 8.468070834423e-11 true resid norm
>
> > > > >
>
> > > > > 1.097968496396e-06 ||r(i)||/||b|| 3.700707484468e-03
>
> > > > >
>
> > > > > 25 KSP Residual norm 7.340678871498e-11
>
> > > > >
>
> > > > > 25 KSP preconditioned resid norm 7.340678871498e-11 true resid norm
>
> > > > >
>
> > > > > 9.388293102378e-07 ||r(i)||/||b|| 3.164328181036e-03
>
> > > > >
>
> > > > > 26 KSP Residual norm 6.452497596580e-11
>
> > > > >
>
> > > > > 26 KSP preconditioned resid norm 6.452497596580e-11 true resid norm
>
> > > > >
>
> > > > > 7.907509627807e-07 ||r(i)||/||b|| 2.665229481464e-03
>
> > > > >
>
> > > > > 27 KSP Residual norm 5.817349095331e-11
>
> > > > >
>
> > > > > 27 KSP preconditioned resid norm 5.817349095331e-11 true resid norm
>
> > > > >
>
> > > > > 6.712549222809e-07 ||r(i)||/||b|| 2.262467568992e-03
>
> > > > >
>
> > > > > 28 KSP Residual norm 5.382715584394e-11
>
> > > > >
>
> > > > > 28 KSP preconditioned resid norm 5.382715584394e-11 true resid norm
>
> > > > >
>
> > > > > 5.883671521302e-07 ||r(i)||/||b|| 1.983093987351e-03
>
> > > > >
>
> > > > > 29 KSP Residual norm 5.072666199584e-11
>
> > > > >
>
> > > > > 29 KSP preconditioned resid norm 5.072666199584e-11 true resid norm
>
> > > > >
>
> > > > > 5.417512216577e-07 ||r(i)||/||b|| 1.825974795533e-03
>
> > > > >
>
> > > > > 30 KSP Residual norm 4.806650672230e-11
>
> > > > >
>
> > > > > 30 KSP preconditioned resid norm 4.806650672230e-11 true resid norm
>
> > > > >
>
> > > > > 5.151866485899e-07 ||r(i)||/||b|| 1.736438789084e-03
>
> > > > >
>
> > > > > 31 KSP Residual norm 4.675001898732e-11
>
> > > > >
>
> > > > > 31 KSP preconditioned resid norm 4.675001898732e-11 true resid norm
>
> > > > >
>
> > > > > 4.825947510614e-07 ||r(i)||/||b|| 1.626587659919e-03
>
> > > > >
>
> > > > > 32 KSP Residual norm 4.530536181592e-11
>
> > > > >
>
> > > > > 32 KSP preconditioned resid norm 4.530536181592e-11 true resid norm
>
> > > > >
>
> > > > > 4.648650267512e-07 ||r(i)||/||b|| 1.566829548764e-03
>
> > > > >
>
> > > > > 33 KSP Residual norm 4.357157235509e-11
>
> > > > >
>
> > > > > 33 KSP preconditioned resid norm 4.357157235509e-11 true resid norm
>
> > > > >
>
> > > > > 4.567209908707e-07 ||r(i)||/||b|| 1.539380041209e-03
>
> > > > >
>
> > > > > 34 KSP Residual norm 4.135715715343e-11
>
> > > > >
>
> > > > > 34 KSP preconditioned resid norm 4.135715715343e-11 true resid norm
>
> > > > >
>
> > > > > 4.469703503011e-07 ||r(i)||/||b|| 1.506515465721e-03
>
> > > > >
>
> > > > > 35 KSP Residual norm 3.851871986267e-11
>
> > > > >
>
> > > > > 35 KSP preconditioned resid norm 3.851871986267e-11 true resid norm
>
> > > > >
>
> > > > > 4.364097171314e-07 ||r(i)||/||b|| 1.470920806730e-03
>
> > > > >
>
> > > > > 36 KSP Residual norm 3.526035391634e-11
>
> > > > >
>
> > > > > 36 KSP preconditioned resid norm 3.526035391634e-11 true resid norm
>
> > > > >
>
> > > > > 4.065382561286e-07 ||r(i)||/||b|| 1.370238920439e-03
>
> > > > >
>
> > > > > 37 KSP Residual norm 3.144895012756e-11
>
> > > > >
>
> > > > > 37 KSP preconditioned resid norm 3.144895012756e-11 true resid norm
>
> > > > >
>
> > > > > 3.810168560499e-07 ||r(i)||/||b|| 1.284218932985e-03
>
> > > > >
>
> > > > > 38 KSP Residual norm 2.735239124106e-11
>
> > > > >
>
> > > > > 38 KSP preconditioned resid norm 2.735239124106e-11 true resid norm
>
> > > > >
>
> > > > > 3.407338934429e-07 ||r(i)||/||b|| 1.148445036278e-03
>
> > > > >
>
> > > > > 39 KSP Residual norm 2.306734554159e-11
>
> > > > >
>
> > > > > 39 KSP preconditioned resid norm 2.306734554159e-11 true resid norm
>
> > > > >
>
> > > > > 2.940484127641e-07 ||r(i)||/||b|| 9.910914251948e-04
>
> > > > >
>
> > > > > 40 KSP Residual norm 1.924842110922e-11
>
> > > > >
>
> > > > > 40 KSP preconditioned resid norm 1.924842110922e-11 true resid norm
>
> > > > >
>
> > > > > 2.443693779909e-07 ||r(i)||/||b|| 8.236480273107e-04
>
> > > > >
>
> > > > > 41 KSP Residual norm 1.618672399310e-11
>
> > > > >
>
> > > > > 41 KSP preconditioned resid norm 1.618672399310e-11 true resid norm
>
> > > > >
>
> > > > > 2.034502467604e-07 ||r(i)||/||b|| 6.857299215548e-04
>
> > > > >
>
> > > > > 42 KSP Residual norm 1.378885267147e-11
>
> > > > >
>
> > > > > 42 KSP preconditioned resid norm 1.378885267147e-11 true resid norm
>
> > > > >
>
> > > > > 1.689714775892e-07 ||r(i)||/||b|| 5.695190834970e-04
>
> > > > >
>
> > > > > 43 KSP Residual norm 1.179601185051e-11
>
> > > > >
>
> > > > > 43 KSP preconditioned resid norm 1.179601185051e-11 true resid norm
>
> > > > >
>
> > > > > 1.483783124701e-07 ||r(i)||/||b|| 5.001097329234e-04
>
> > > > >
>
> > > > > 44 KSP Residual norm 9.928908801463e-12
>
> > > > >
>
> > > > > 44 KSP preconditioned resid norm 9.928908801463e-12 true resid norm
>
> > > > >
>
> > > > > 1.276885725440e-07 ||r(i)||/||b|| 4.303748765523e-04
>
> > > > >
>
> > > > > 45 KSP Residual norm 8.512555069524e-12
>
> > > > >
>
> > > > > 45 KSP preconditioned resid norm 8.512555069524e-12 true resid norm
>
> > > > >
>
> > > > > 1.020939171370e-07 ||r(i)||/||b|| 3.441079817022e-04
>
> > > > >
>
> > > > > 46 KSP Residual norm 7.600214409478e-12
>
> > > > >
>
> > > > > 46 KSP preconditioned resid norm 7.600214409478e-12 true resid norm
>
> > > > >
>
> > > > > 8.873369034852e-08 ||r(i)||/||b|| 2.990772805186e-04
>
> > > > >
>
> > > > > 47 KSP Residual norm 6.577382322417e-12
>
> > > > >
>
> > > > > 47 KSP preconditioned resid norm 6.577382322417e-12 true resid norm
>
> > > > >
>
> > > > > 8.285748008098e-08 ||r(i)||/||b|| 2.792714888326e-04
>
> > > > >
>
> > > > > 48 KSP Residual norm 5.701374824383e-12
>
> > > > >
>
> > > > > 48 KSP preconditioned resid norm 5.701374824383e-12 true resid norm
>
> > > > >
>
> > > > > 6.754179947516e-08 ||r(i)||/||b|| 2.276499222452e-04
>
> > > > >
>
> > > > > 49 KSP Residual norm 5.201909133133e-12
>
> > > > >
>
> > > > > 49 KSP preconditioned resid norm 5.201909133133e-12 true resid norm
>
> > > > >
>
> > > > > 5.807496359589e-08 ||r(i)||/||b|| 1.957419116714e-04
>
> > > > >
>
> > > > > 50 KSP Residual norm 4.803415857522e-12
>
> > > > >
>
> > > > > 50 KSP preconditioned resid norm 4.803415857522e-12 true resid norm
>
> > > > >
>
> > > > > 5.648916764539e-08 ||r(i)||/||b|| 1.903969796791e-04
>
> > > > >
>
> > > > > 51 KSP Residual norm 4.213404847970e-12
>
> > > > >
>
> > > > > 51 KSP preconditioned resid norm 4.213404847970e-12 true resid norm
>
> > > > >
>
> > > > > 5.312367187239e-08 ||r(i)||/||b|| 1.790535618699e-04
>
> > > > >
>
> > > > > 52 KSP Residual norm 3.583723287544e-12
>
> > > > >
>
> > > > > 52 KSP preconditioned resid norm 3.583723287544e-12 true resid norm
>
> > > > >
>
> > > > > 4.482581392980e-08 ||r(i)||/||b|| 1.510855964009e-04
>
> > > > >
>
> > > > > 53 KSP Residual norm 3.055433343791e-12
>
> > > > >
>
> > > > > 53 KSP preconditioned resid norm 3.055433343791e-12 true resid norm
>
> > > > >
>
> > > > > 3.733365588478e-08 ||r(i)||/||b|| 1.258332458617e-04
>
> > > > >
>
> > > > > 54 KSP Residual norm 2.632415195178e-12
>
> > > > >
>
> > > > > 54 KSP preconditioned resid norm 2.632415195178e-12 true resid norm
>
> > > > >
>
> > > > > 3.259461418510e-08 ||r(i)||/||b|| 1.098602856676e-04
>
> > > > >
>
> > > > > 55 KSP Residual norm 2.358953414259e-12
>
> > > > >
>
> > > > > 55 KSP preconditioned resid norm 2.358953414259e-12 true resid norm
>
> > > > >
>
> > > > > 2.755891784400e-08 ||r(i)||/||b|| 9.288744974367e-05
>
> > > > >
>
> > > > > 56 KSP Residual norm 2.131084837625e-12
>
> > > > >
>
> > > > > 56 KSP preconditioned resid norm 2.131084837625e-12 true resid norm
>
> > > > >
>
> > > > > 2.435335186857e-08 ||r(i)||/||b|| 8.208307599694e-05
>
> > > > >
>
> > > > > 57 KSP Residual norm 1.906113965754e-12
>
> > > > >
>
> > > > > 57 KSP preconditioned resid norm 1.906113965754e-12 true resid norm
>
> > > > >
>
> > > > > 2.234444331024e-08 ||r(i)||/||b|| 7.531204116139e-05
>
> > > > >
>
> > > > > 58 KSP Residual norm 1.666029207693e-12
>
> > > > >
>
> > > > > 58 KSP preconditioned resid norm 1.666029207693e-12 true resid norm
>
> > > > >
>
> > > > > 2.187608862971e-08 ||r(i)||/||b|| 7.373344971974e-05
>
> > > > >
>
> > > > > 59 KSP Residual norm 1.397594452516e-12
>
> > > > >
>
> > > > > 59 KSP preconditioned resid norm 1.397594452516e-12 true resid norm
>
> > > > >
>
> > > > > 1.884281030995e-08 ||r(i)||/||b|| 6.350977224879e-05
>
> > > > >
>
> > > > > 60 KSP Residual norm 1.000720808126e-12
>
> > > > >
>
> > > > > 60 KSP preconditioned resid norm 1.000720808126e-12 true resid norm
>
> > > > >
>
> > > > > 1.495264965711e-08 ||r(i)||/||b|| 5.039796923166e-05
>
> > > > >
>
> > > > > 61 KSP Residual norm 8.368281175845e-13
>
> > > > >
>
> > > > > 61 KSP preconditioned resid norm 8.368281175845e-13 true resid norm
>
> > > > >
>
> > > > > 1.198110007063e-08 ||r(i)||/||b|| 4.038234871863e-05
>
> > > > >
>
> > > > > 62 KSP Residual norm 6.975866756145e-13
>
> > > > >
>
> > > > > 62 KSP preconditioned resid norm 6.975866756145e-13 true resid norm
>
> > > > >
>
> > > > > 8.713139114636e-09 ||r(i)||/||b|| 2.936767242465e-05
>
> > > > >
>
> > > > > 63 KSP Residual norm 6.055685829863e-13
>
> > > > >
>
> > > > > 63 KSP preconditioned resid norm 6.055685829863e-13 true resid norm
>
> > > > >
>
> > > > > 7.208696201852e-09 ||r(i)||/||b|| 2.429694119186e-05
>
> > > > >
>
> > > > > 64 KSP Residual norm 5.321880026091e-13
>
> > > > >
>
> > > > > 64 KSP preconditioned resid norm 5.321880026091e-13 true resid norm
>
> > > > >
>
> > > > > 6.450668328122e-09 ||r(i)||/||b|| 2.174200502115e-05
>
> > > > >
>
> > > > > 65 KSP Residual norm 4.652520314729e-13
>
> > > > >
>
> > > > > 65 KSP preconditioned resid norm 4.652520314729e-13 true resid norm
>
> > > > >
>
> > > > > 5.767198491137e-09 ||r(i)||/||b|| 1.943836703022e-05
>
> > > > >
>
> > > > > 66 KSP Residual norm 4.095554125778e-13
>
> > > > >
>
> > > > > 66 KSP preconditioned resid norm 4.095554125778e-13 true resid norm
>
> > > > >
>
> > > > > 5.117612129931e-09 ||r(i)||/||b|| 1.724893343844e-05
>
> > > > >
>
> > > > > 67 KSP Residual norm 3.575261581129e-13
>
> > > > >
>
> > > > > 67 KSP preconditioned resid norm 3.575261581129e-13 true resid norm
>
> > > > >
>
> > > > > 4.479181564454e-09 ||r(i)||/||b|| 1.509710050359e-05
>
> > > > >
>
> > > > > 68 KSP Residual norm 3.109053579360e-13
>
> > > > >
>
> > > > > 68 KSP preconditioned resid norm 3.109053579360e-13 true resid norm
>
> > > > >
>
> > > > > 3.779777902208e-09 ||r(i)||/||b|| 1.273975748689e-05
>
> > > > >
>
> > > > > 69 KSP Residual norm 2.721884401030e-13
>
> > > > >
>
> > > > > 69 KSP preconditioned resid norm 2.721884401030e-13 true resid norm
>
> > > > >
>
> > > > > 3.393605957929e-09 ||r(i)||/||b|| 1.143816330711e-05
>
> > > > >
>
> > > > > Linear solve converged due to CONVERGED_RTOL iterations 69
>
> > > > >
>
> > > > > KSP Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: gmres
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Orthogonalization with no iterative refinement
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > GMRES: happy breakdown tolerance 1e-30
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > maximum iterations=10000, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using PRECONDITIONED norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: ilu
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > ILU: out-of-place factorization
>
> > > > >
>
> > > > > 0 levels of fill
>
> > > > >
>
> > > > > tolerance for zero pivot 2.22045e-14
>
> > > > >
>
> > > > > matrix ordering: natural
>
> > > > >
>
> > > > > factor fill ratio given 1, needed 1
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Factored matrix follows:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=16384, cols=16384
>
> > > > >
>
> > > > > package used to perform factorization: petsc
>
> > > > >
>
> > > > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=16384, cols=16384
>
> > > > >
>
> > > > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > has attached null space
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > 4)
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Linear solve converged due to CONVERGED_ITS iterations 1
>
> > > > >
>
> > > > > KSP Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: preonly
>
> > > > >
>
> > > > > maximum iterations=10000, initial guess is zero
>
> > > > >
>
> > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>
> > > > >
>
> > > > > left preconditioning
>
> > > > >
>
> > > > > using NONE norm type for convergence test
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > PC Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: lu
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > LU: out-of-place factorization
>
> > > > >
>
> > > > > tolerance for zero pivot 2.22045e-14
>
> > > > >
>
> > > > > matrix ordering: nd
>
> > > > >
>
> > > > > factor fill ratio given 5, needed 9.6582
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > Factored matrix follows:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=16384, cols=16384
>
> > > > >
>
> > > > > package used to perform factorization: petsc
>
> > > > >
>
> > > > > total: nonzeros=791200, allocated nonzeros=791200
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > linear system matrix = precond matrix:
>
> > > > >
>
> > > > > Mat Object: 1 MPI processes
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > type: seqaij
>
> > > > >
>
> > > > > rows=16384, cols=16384
>
> > > > >
>
> > > > > total: nonzeros=81920, allocated nonzeros=81920
>
> > > > >
>
> > > > > total number of mallocs used during MatSetValues calls =0
>
> > > > >
>
> > > > >
>
> > > > >
>
> > > > > has attached null space
>
> > > > >
>
> > > > > not using I-node routines
>
> > > > >
>
> > > > > On Tuesday 01 September 2015 03:40:14 Matthew Knepley wrote:
>
> > > > > > On Tue, Sep 1, 2015 at 2:22 AM, Filippo Leonardi <
>
> > > > > >
>
> > > > > > filippo.leonardi at sam.math.ethz.ch> wrote:
>
> > > > > > > Dear PETSc Users,
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > > I want to use multigrid to solve uniform (just Laplace)
> poisson in
>
> > > > >
>
> > > > > 2D/3D
>
> > > > >
>
> > > > > > > on cartesian, non-uniform meshes with a standard 5 (7)-points
>
> > >
>
> > > stencil
>
> > >
>
> > > > > FD.
>
> > > > >
>
> > > > > > > I always scaled my Poisson matrix like in the doc examples,
> i.e.
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > > multiplying by dx*dy (so that in ComputeRHS I need to scale b,
> in
>
> > >
>
> > > A*x
>
> > >
>
> > > > > = b,
>
> > > > >
>
> > > > > > > as well). This always worked properly with both MG/GAMG and
> with
>
> > > > >
>
> > > > > galerkin
>
> > > > >
>
> > > > > > > matrices.
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > > Now I'd like to use non-uniform meshes, therefore the scaling
> is
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > > non-uniform. However I cannot get my matrices to scale properly
>
> > >
>
> > > with
>
> > >
>
> > > > > any
>
> > > > >
>
> > > > > > > sort of multigrid.
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > > One would think that without scaling, i.e. solving the original
>
> > > > >
>
> > > > > system, at
>
> > > > >
>
> > > > > > > least MG+galerkin or GAMG should work anyways provided the
> matrix
>
> > > > > > > A
>
> > > > >
>
> > > > > and b
>
> > > > >
>
> > > > > > > are consistent.
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > > I tried without boundaries (i.e. torus), so this is not the
>
> > >
>
> > > problem.
>
> > >
>
> > > > > > The point of that scaling is to make the boundary values the same
>
> > >
>
> > > size
>
> > >
>
> > > > > > as
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > > the residuals. If you have
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > > no boundary values, the problem is scale invariant since its
> linear.
>
> > >
>
> > > It
>
> > >
>
> > > > > > would be nice to scale it so everything
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > > is about size 1.
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > > When asking about convergence always run with -ksp_view
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > > -ksp_monitor_true_residual -ksp_converged_reason
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > > Thanks,
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > >
>
> > > > > > Matt
>
> > > > > >
>
> > > > > > > Anyone did/knows how to do this properly?
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > > Thanks,
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > >
>
> > > > > > > F
>
>
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/daf679a1/attachment-0001.html>

From filippo.leon at gmail.com  Tue Sep  1 06:53:00 2015
From: filippo.leon at gmail.com (Filippo Leonardi)
Date: Tue, 01 Sep 2015 13:53:00 +0200
Subject: [petsc-users] 2D Poisson on nonuniform meshes
In-Reply-To: <CAMYG4GncTzqDzMe3s86e6a0A7RsZhGT7F-p_sqjgB_mOT-cL0Q@mail.gmail.com>
References: <3634751.Hfn5OcL7V3@besikowitch-iii>
	<2620235.Qtx0xDWqJW@besikowitch-iii>
	<CAMYG4GncTzqDzMe3s86e6a0A7RsZhGT7F-p_sqjgB_mOT-cL0Q@mail.gmail.com>
Message-ID: <2124074.mjk2bZhIH0@besikowitch-iii>

On Tuesday 01 September 2015 06:32:38 you wrote:
> > > 
> > > 1) The KSP view does not say it is shifting. Are you using the latest
> > > 
> > > release?
> > 
> > yes, 3.6. Does PETSc warn for that even if I set the nullspace? I can also
> > check MUMPS or something else.
> 
> I am not sure what you think PETSc does here. If shifting were enabled, it
> would add some
> diagonal matrix to the input matrix and continue the factorization. This
> would mean that the
> factors were not, in fact, a factorization of the input matrix, and you
> would not get the exact
> solution in one iterate.

I though PETSc would've replace my pivots with small eps, which is actually not a problem in my 
case

> 
> > > 2) If it shifted, it would not solve in a single iterate.
> > 
> > even with preonly?
> 
> You would have a large residual. Do you?

Actually, I get a perfect solution.

> 
> > > 3) Your GAMG results imply that something is wrong with the coarse
> > > solve.
> > > 
> > > This is exactly what would happen if
> > > 
> > > that problem was not solved accurately (its off by > 10 orders of
> > > 
> > > magnitude).
> > 
> > yes, but GAMG builds is own coarse solvers so either the problem is
> > already in the definition of A and b (likely) or it is a bug in gamg.
> 
> Yes. GAMG uses the constants to build the basis, on the assumption that
> they are in the (near) nullspace of the
> operator with no boundary conditions. Since this is far off, I think this
> must not be true for your A.

> 
> > > It sounds like your operator is not singular, and its not the Laplacian
> > > 
> > > since it does not look like the Neumann version
> > > 
> > > has constants as a null space.
> > 
> > I'm using periodic boundaries, and constants are in kern(A)
> 
> Did you check?

Checked with VecSet and MatMult just in case, I get a machine eps constant vector.

> 
>    Matt
> 
> > Thanks a lot for your time.
> > 
> > > Thanks,
> > > 
> > > 
> > > > > > > > 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/f881e4c4/attachment.html>

From knepley at gmail.com  Tue Sep  1 12:13:34 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 Sep 2015 12:13:34 -0500
Subject: [petsc-users] 2D Poisson on nonuniform meshes
In-Reply-To: <2124074.mjk2bZhIH0@besikowitch-iii>
References: <3634751.Hfn5OcL7V3@besikowitch-iii>
	<2620235.Qtx0xDWqJW@besikowitch-iii>
	<CAMYG4GncTzqDzMe3s86e6a0A7RsZhGT7F-p_sqjgB_mOT-cL0Q@mail.gmail.com>
	<2124074.mjk2bZhIH0@besikowitch-iii>
Message-ID: <CAMYG4GkVnj4tJD5gdeUiOLxgKOUhDkg16nM6MpB7yRc3QeSSAw@mail.gmail.com>

On Tue, Sep 1, 2015 at 6:53 AM, Filippo Leonardi <filippo.leon at gmail.com>
wrote:

> On Tuesday 01 September 2015 06:32:38 you wrote:
>
> > > >
>
> > > > 1) The KSP view does not say it is shifting. Are you using the latest
>
> > > >
>
> > > > release?
>
> > >
>
> > > yes, 3.6. Does PETSc warn for that even if I set the nullspace? I can
> also
>
> > > check MUMPS or something else.
>
> >
>
> > I am not sure what you think PETSc does here. If shifting were enabled,
> it
>
> > would add some
>
> > diagonal matrix to the input matrix and continue the factorization. This
>
> > would mean that the
>
> > factors were not, in fact, a factorization of the input matrix, and you
>
> > would not get the exact
>
> > solution in one iterate.
>
>
>
> I though PETSc would've replace my pivots with small eps, which is
> actually not a problem in my case
>
>
>
> >
>
> > > > 2) If it shifted, it would not solve in a single iterate.
>
> > >
>
> > > even with preonly?
>
> >
>
> > You would have a large residual. Do you?
>
>
>
> Actually, I get a perfect solution.
>
>
>
> >
>
> > > > 3) Your GAMG results imply that something is wrong with the coarse
>
> > > > solve.
>
> > > >
>
> > > > This is exactly what would happen if
>
> > > >
>
> > > > that problem was not solved accurately (its off by > 10 orders of
>
> > > >
>
> > > > magnitude).
>
> > >
>
> > > yes, but GAMG builds is own coarse solvers so either the problem is
>
> > > already in the definition of A and b (likely) or it is a bug in gamg.
>
> >
>
> > Yes. GAMG uses the constants to build the basis, on the assumption that
>
> > they are in the (near) nullspace of the
>
> > operator with no boundary conditions. Since this is far off, I think this
>
> > must not be true for your A.
>
>
>
> >
>
> > > > It sounds like your operator is not singular, and its not the
> Laplacian
>
> > > >
>
> > > > since it does not look like the Neumann version
>
> > > >
>
> > > > has constants as a null space.
>
> > >
>
> > > I'm using periodic boundaries, and constants are in kern(A)
>
> >
>
> > Did you check?
>
>
>
> Checked with VecSet and MatMult just in case, I get a machine eps constant
> vector.
>

Okay, it seems that we have the best chance of figuring out the problem by
looking at option 2) and 4).

First, is your rhs consistent, meaning is it orthogonal to your nullspace?

   Matt


> >
>
> > Matt
>
> >
>
> > > Thanks a lot for your time.
>
> > >
>
> > > > Thanks,
>
> > > >
>
> > > >
>
> > > > > > > > >
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/a09d7cae/attachment.html>

From bsmith at mcs.anl.gov  Tue Sep  1 13:48:34 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 1 Sep 2015 13:48:34 -0500
Subject: [petsc-users] Integrating TAO into SNES and TS
In-Reply-To: <CAP2=TMiJUz6yM5vMjThMTgZ6CmRsS+da6dU+4FuJq8qk6c+fdw@mail.gmail.com>
References: <CAP2=TMgF8fKyX0o--rRz+cw01=d5ws6iCZu=2PVq1qBJq50okA@mail.gmail.com>
	<16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov>
	<CAP2=TMhD41MWMf=n+Z7ZdDdNYvRVx91fL956AXSjtJb_vz01gg@mail.gmail.com>
	<0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov>
	<CAP2=TMh=NSnObLKwgR_ZJHZDqNe_c+VO4NPkSFnA3X+NLd0Amg@mail.gmail.com>
	<6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov>
	<CAP2=TMiZXce6FBKMRNBtnyn3Ug7cnMn_2ZhgR2547=OqpO8Nog@mail.gmail.com>
	<C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
	<CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>
	<CAMYG4G=MTDKpRrfnV+nBZvCGWNhTe-qtSkmVoCOF4J_uLS=e4A@mail.gmail.com>
	<CAP2=TMjdp3iKJ8943dHwKQmFg1cb9WOt29MgEgaiNTmMMcmoow@mail.gmail.com>
	<CAMYG4GnvWpxNExr_VzkVt4q5UAEwMcHXMSYyahP9gxpP+-XRbw@mail.gmail.com>
	<CAP2=TMiJUz6yM5vMjThMTgZ6CmRsS+da6dU+4FuJq8qk6c+fdw@mail.gmail.com>
Message-ID: <AB01DEC6-517D-48EE-AB00-504E42E9D5F1@mcs.anl.gov>


   I think we are talking past each other.

   With bound constraint VI's there is no optimization, there is an equation F(x) = 0 (where F may be linear or nonlinear) and constraints  a <= x <= c. With VIs the equation F_i(x) is simply not satisfied if x_i is on a bound (that is x_i = a_i or x_i = b_i),

   With optimization if you have an equality constraint and inequality constraints; if to satisfy an inequality constraint FORCES an equality constraint to not be satisfied then the constraints are not compatible and the problem isn't properly posed.

   Barry


> On Sep 1, 2015, at 4:15 AM, Justin Chang <jychang48 at gmail.com> wrote:
> 
> But if I add those linear equality constraint equations to my original problem, would they not be satisfied anyway? Say I add this to my weak form:
> 
> Ax = b
> 
> But once i subject x to some bounded constraints, Ax != b. Unless I add some sort of penalty where extra weighting is added to this property...
> 
> On Tue, Sep 1, 2015 at 3:02 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Sep 1, 2015 at 3:46 AM, Justin Chang <jychang48 at gmail.com> wrote:
> I would like to simultaneously enforce both discrete maximum principle and local mass/species balance. Because even if a locally conservative scheme like RT0 is used, as soon as these bounded constraints are applied, i lose the mass balance.
> 
> What I am saying is, can't you just add "linear equality constraints" as more equations?
> 
>   Thanks,
> 
>     Matt
>  
> On Tue, Sep 1, 2015 at 2:33 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Sep 1, 2015 at 3:11 AM, Justin Chang <jychang48 at gmail.com> wrote:
> Barry,
> 
> That's good to know thanks.
> 
> On a related note, is it possible for VI to one day include linear equality constraints?
> 
> How are these different from just using more equations?
> 
>   Thanks,
> 
>     Matt
>  
> Thanks,
> Justin
> 
> On Mon, Aug 31, 2015 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Aug 31, 2015, at 7:36 PM, Justin Chang <jychang48 at gmail.com> wrote:
> >
> > Coming back to this,
> >
> > Say I now want to ensure the DMP for advection-diffusion equations. The linear operator is now asymmetric and non-self-adjoint (assuming I do something like SUPG or finite volume), meaning I cannot simply solve this problem without any manipulation (e.g. normalizing the equations) using TAO's optimization solvers. Does this statement also hold true for SNESVI?
> 
>   SNESVI doesn't care about symmetry etc
> 
> >
> > Thanks,
> > Justin
> >
> > On Fri, Apr 3, 2015 at 7:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Apr 3, 2015, at 7:35 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > >
> > > I guess I will have to write my own code then :)
> > >
> > > I am not all that familiar with Variational Inequalities at the moment, but if my Jacobian is symmetric and positive definite and I only have lower and upper bounds, doesn't the problem simply reduce to that of a convex optimization? That is, with SNES act as if it were Tao?
> >
> >   Yes, I think that is essentially correctly.
> >
> >   Barry
> >
> > >
> > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >
> > >   Justin,
> > >
> > >    We haven't done anything with TS to handle variational inequalities. So you can either write your own backward Euler (outside of TS) that solves each time-step problem either as 1) an optimization problem using Tao or 2) as a variational inequality using SNES.
> > >
> > >    More adventurously you could look at the TSTHETA code in TS (which is a general form that includes Euler, Backward Euler and Crank-Nicolson and see if you can add the constraints to the SNES problem that is solved; in theory this is straightforward but it would require understanding the current code (which Jed, of course, overwrote :-). I think you should do this.
> > >
> > >   Barry
> > >
> > >
> > > > On Apr 3, 2015, at 12:31 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > >
> > > > I am solving the following anisotropic transient diffusion equation subject to 0 bounds:
> > > >
> > > > du/dt = div[D*grad[u]] + f
> > > >
> > > > Where the dispersion tensor D(x) is symmetric and positive definite. This formulation violates the discrete maximum principles so one of the ways to ensure nonnegative concentrations is to employ convex optimization. I am following the procedures in Nakshatrala and Valocchi (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF.
> > > >
> > > > The Variational Inequality method works gives what I want for my transient case, but what if I want to implement the Tao methodology in TS? That is, what TS functions do I need to set up steps a) through e) for each time step (also the Jacobian remains the same for all time steps so I would only call this once). Normally I would just call TSSolve() and let the libraries and functions do everything, but I would like to incorporate TaoSolve into every time step.
> > > >
> > > > Thanks,
> > > >
> > > > --
> > > > Justin Chang
> > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > University of Houston, Department of Civil and Environmental Engineering
> > > > Houston, TX 77004
> > > > (512) 963-3262
> > > >
> > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > >
> > > >   An alternative approach is for you to solve it as a (non)linear variational inequality. See src/snes/examples/tutorials/ex9.c
> > > >
> > > >   How you should proceed depends on your long term goal. What problem do you really want to solve? Is it really a linear time dependent problem with 0 bounds on U? Can the problem always be represented as an optimization problem easily? What are  and what will be the properties of K? For example if K is positive definite then likely the bounds will remain try without explicitly providing the constraints.
> > > >
> > > >   Barry
> > > >
> > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I have a two part question regarding the integration of the following optimization problem
> > > > >
> > > > > min 1/2 u^T*K*u + u^T*f
> > > > > S.T. u >= 0
> > > > >
> > > > > into SNES and TS
> > > > >
> > > > > 1) For SNES, assuming I am working with a linear FE equation, I have the following algorithm/steps for solving my problem
> > > > >
> > > > > a) Set an initial guess x
> > > > > b) Obtain residual r and jacobian A through functions SNESComputeFunction() and SNESComputeJacobian() respectively
> > > > > c) Form vector b = r - A*x
> > > > > d) Set Hessian equal to A, gradient to A*x, objective function value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector
> > > > > e) Call TaoSolve
> > > > >
> > > > > This works well at the moment, but my question is there a more "efficient" way of doing this? Because with my current setup, I am making a rather bold assumption that my problem would converge in one SNES iteration without the bounded constraints and does not have any unexpected nonlinearities.
> > > > >
> > > > > 2) How would I go about doing the above for time-stepping problems? At each time step, I want to solve a convex optimization subject to the lower bounds constraint. I plan on using backward euler and my resulting jacobian should still be compatible with the above optimization problem.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > --
> > > > > Justin Chang
> > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > University of Houston, Department of Civil and Environmental Engineering
> > > > > Houston, TX 77004
> > > > > (512) 963-3262
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Justin Chang
> > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > University of Houston, Department of Civil and Environmental Engineering
> > > > Houston, TX 77004
> > > > (512) 963-3262
> > >
> > >
> > >
> > >
> > > --
> > > Justin Chang
> > > PhD Candidate, Civil Engineering - Computational Sciences
> > > University of Houston, Department of Civil and Environmental Engineering
> > > Houston, TX 77004
> > > (512) 963-3262
> >
> >
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 


From bsmith at mcs.anl.gov  Tue Sep  1 14:01:27 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 1 Sep 2015 14:01:27 -0500
Subject: [petsc-users] 2D Poisson on nonuniform meshes
In-Reply-To: <3634751.Hfn5OcL7V3@besikowitch-iii>
References: <3634751.Hfn5OcL7V3@besikowitch-iii>
Message-ID: <5D732EA1-4236-47F8-9E30-C359021F020F@mcs.anl.gov>


   You need to get the nonuniform finite difference stencil right and the refinement right and the interpolation right. For example when you refine are you putting the new grid points 1/2 way between the coarser 2 or defined by the "grading" of the coarse mesh.  I recommend doing the problem with 1d to understand exactly what you are doing, get it working there. Once you understand it doing 2 or 3d is easy.

  Barry



> On Sep 1, 2015, at 2:22 AM, Filippo Leonardi <filippo.leonardi at sam.math.ethz.ch> wrote:
> 
> Dear PETSc Users,
>  
> I want to use multigrid to solve uniform (just Laplace) poisson in 2D/3D on cartesian, non-uniform meshes with a standard 5 (7)-points stencil FD.
>  
> I always scaled my Poisson matrix like in the doc examples, i.e. multiplying by dx*dy (so that in ComputeRHS I need to scale b, in A*x = b, as well). This always worked properly with both MG/GAMG and with galerkin matrices.
>  
> Now I'd like to use non-uniform meshes, therefore the scaling is non-uniform. However I cannot get my matrices to scale properly with any sort of multigrid.
>  
> One would think that without scaling, i.e. solving the original system, at least MG+galerkin or GAMG should work anyways provided the matrix A and b are consistent.
>  
> I tried without boundaries (i.e. torus), so this is not the problem.
>  
> Anyone did/knows how to do this properly?
>  
> Thanks,
> F


From jychang48 at gmail.com  Tue Sep  1 17:14:38 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 1 Sep 2015 16:14:38 -0600
Subject: [petsc-users] Integrating TAO into SNES and TS
In-Reply-To: <AB01DEC6-517D-48EE-AB00-504E42E9D5F1@mcs.anl.gov>
References: <CAP2=TMgF8fKyX0o--rRz+cw01=d5ws6iCZu=2PVq1qBJq50okA@mail.gmail.com>
	<16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov>
	<CAP2=TMhD41MWMf=n+Z7ZdDdNYvRVx91fL956AXSjtJb_vz01gg@mail.gmail.com>
	<0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov>
	<CAP2=TMh=NSnObLKwgR_ZJHZDqNe_c+VO4NPkSFnA3X+NLd0Amg@mail.gmail.com>
	<6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov>
	<CAP2=TMiZXce6FBKMRNBtnyn3Ug7cnMn_2ZhgR2547=OqpO8Nog@mail.gmail.com>
	<C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
	<CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>
	<CAMYG4G=MTDKpRrfnV+nBZvCGWNhTe-qtSkmVoCOF4J_uLS=e4A@mail.gmail.com>
	<CAP2=TMjdp3iKJ8943dHwKQmFg1cb9WOt29MgEgaiNTmMMcmoow@mail.gmail.com>
	<CAMYG4GnvWpxNExr_VzkVt4q5UAEwMcHXMSYyahP9gxpP+-XRbw@mail.gmail.com>
	<CAP2=TMiJUz6yM5vMjThMTgZ6CmRsS+da6dU+4FuJq8qk6c+fdw@mail.gmail.com>
	<AB01DEC6-517D-48EE-AB00-504E42E9D5F1@mcs.anl.gov>
Message-ID: <CAP2=TMhAipXm20+Ph5Qd=NT6JRp9Vb8AC2DVojXzh4uu1ynhEQ@mail.gmail.com>

So in VI, the i-th component/cell/point of x is "corrected" if it is below
lb_i or above ub_i. All other x_i that satisfy their respective bounds
remain untouched. Hence only the corrected components will violated the
original equation

In optimization, every single x_i will be "shifted" such that every single
x_i meets lb_i and ub_i. Hence it's possible all components will violate
the original equation

Is that the general idea here?

On Tue, Sep 1, 2015 at 12:48 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    I think we are talking past each other.
>
>    With bound constraint VI's there is no optimization, there is an
> equation F(x) = 0 (where F may be linear or nonlinear) and constraints  a
> <= x <= c. With VIs the equation F_i(x) is simply not satisfied if x_i is
> on a bound (that is x_i = a_i or x_i = b_i),
>
>    With optimization if you have an equality constraint and inequality
> constraints; if to satisfy an inequality constraint FORCES an equality
> constraint to not be satisfied then the constraints are not compatible and
> the problem isn't properly posed.
>
>    Barry
>
>
> > On Sep 1, 2015, at 4:15 AM, Justin Chang <jychang48 at gmail.com> wrote:
> >
> > But if I add those linear equality constraint equations to my original
> problem, would they not be satisfied anyway? Say I add this to my weak form:
> >
> > Ax = b
> >
> > But once i subject x to some bounded constraints, Ax != b. Unless I add
> some sort of penalty where extra weighting is added to this property...
> >
> > On Tue, Sep 1, 2015 at 3:02 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Tue, Sep 1, 2015 at 3:46 AM, Justin Chang <jychang48 at gmail.com>
> wrote:
> > I would like to simultaneously enforce both discrete maximum principle
> and local mass/species balance. Because even if a locally conservative
> scheme like RT0 is used, as soon as these bounded constraints are applied,
> i lose the mass balance.
> >
> > What I am saying is, can't you just add "linear equality constraints" as
> more equations?
> >
> >   Thanks,
> >
> >     Matt
> >
> > On Tue, Sep 1, 2015 at 2:33 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Tue, Sep 1, 2015 at 3:11 AM, Justin Chang <jychang48 at gmail.com>
> wrote:
> > Barry,
> >
> > That's good to know thanks.
> >
> > On a related note, is it possible for VI to one day include linear
> equality constraints?
> >
> > How are these different from just using more equations?
> >
> >   Thanks,
> >
> >     Matt
> >
> > Thanks,
> > Justin
> >
> > On Mon, Aug 31, 2015 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Aug 31, 2015, at 7:36 PM, Justin Chang <jychang48 at gmail.com> wrote:
> > >
> > > Coming back to this,
> > >
> > > Say I now want to ensure the DMP for advection-diffusion equations.
> The linear operator is now asymmetric and non-self-adjoint (assuming I do
> something like SUPG or finite volume), meaning I cannot simply solve this
> problem without any manipulation (e.g. normalizing the equations) using
> TAO's optimization solvers. Does this statement also hold true for SNESVI?
> >
> >   SNESVI doesn't care about symmetry etc
> >
> > >
> > > Thanks,
> > > Justin
> > >
> > > On Fri, Apr 3, 2015 at 7:38 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > > > On Apr 3, 2015, at 7:35 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > >
> > > > I guess I will have to write my own code then :)
> > > >
> > > > I am not all that familiar with Variational Inequalities at the
> moment, but if my Jacobian is symmetric and positive definite and I only
> have lower and upper bounds, doesn't the problem simply reduce to that of a
> convex optimization? That is, with SNES act as if it were Tao?
> > >
> > >   Yes, I think that is essentially correctly.
> > >
> > >   Barry
> > >
> > > >
> > > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > >
> > > >   Justin,
> > > >
> > > >    We haven't done anything with TS to handle variational
> inequalities. So you can either write your own backward Euler (outside of
> TS) that solves each time-step problem either as 1) an optimization problem
> using Tao or 2) as a variational inequality using SNES.
> > > >
> > > >    More adventurously you could look at the TSTHETA code in TS
> (which is a general form that includes Euler, Backward Euler and
> Crank-Nicolson and see if you can add the constraints to the SNES problem
> that is solved; in theory this is straightforward but it would require
> understanding the current code (which Jed, of course, overwrote :-). I
> think you should do this.
> > > >
> > > >   Barry
> > > >
> > > >
> > > > > On Apr 3, 2015, at 12:31 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > > >
> > > > > I am solving the following anisotropic transient diffusion
> equation subject to 0 bounds:
> > > > >
> > > > > du/dt = div[D*grad[u]] + f
> > > > >
> > > > > Where the dispersion tensor D(x) is symmetric and positive
> definite. This formulation violates the discrete maximum principles so one
> of the ways to ensure nonnegative concentrations is to employ convex
> optimization. I am following the procedures in Nakshatrala and Valocchi
> (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF.
> > > > >
> > > > > The Variational Inequality method works gives what I want for my
> transient case, but what if I want to implement the Tao methodology in TS?
> That is, what TS functions do I need to set up steps a) through e) for each
> time step (also the Jacobian remains the same for all time steps so I would
> only call this once). Normally I would just call TSSolve() and let the
> libraries and functions do everything, but I would like to incorporate
> TaoSolve into every time step.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > --
> > > > > Justin Chang
> > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > > Houston, TX 77004
> > > > > (512) 963-3262
> > > > >
> > > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > >
> > > > >   An alternative approach is for you to solve it as a (non)linear
> variational inequality. See src/snes/examples/tutorials/ex9.c
> > > > >
> > > > >   How you should proceed depends on your long term goal. What
> problem do you really want to solve? Is it really a linear time dependent
> problem with 0 bounds on U? Can the problem always be represented as an
> optimization problem easily? What are  and what will be the properties of
> K? For example if K is positive definite then likely the bounds will remain
> try without explicitly providing the constraints.
> > > > >
> > > > >   Barry
> > > > >
> > > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang <jchang27 at uh.edu>
> wrote:
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I have a two part question regarding the integration of the
> following optimization problem
> > > > > >
> > > > > > min 1/2 u^T*K*u + u^T*f
> > > > > > S.T. u >= 0
> > > > > >
> > > > > > into SNES and TS
> > > > > >
> > > > > > 1) For SNES, assuming I am working with a linear FE equation, I
> have the following algorithm/steps for solving my problem
> > > > > >
> > > > > > a) Set an initial guess x
> > > > > > b) Obtain residual r and jacobian A through functions
> SNESComputeFunction() and SNESComputeJacobian() respectively
> > > > > > c) Form vector b = r - A*x
> > > > > > d) Set Hessian equal to A, gradient to A*x, objective function
> value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector
> > > > > > e) Call TaoSolve
> > > > > >
> > > > > > This works well at the moment, but my question is there a more
> "efficient" way of doing this? Because with my current setup, I am making a
> rather bold assumption that my problem would converge in one SNES iteration
> without the bounded constraints and does not have any unexpected
> nonlinearities.
> > > > > >
> > > > > > 2) How would I go about doing the above for time-stepping
> problems? At each time step, I want to solve a convex optimization subject
> to the lower bounds constraint. I plan on using backward euler and my
> resulting jacobian should still be compatible with the above optimization
> problem.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > --
> > > > > > Justin Chang
> > > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > > > Houston, TX 77004
> > > > > > (512) 963-3262
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Justin Chang
> > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > > Houston, TX 77004
> > > > > (512) 963-3262
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Justin Chang
> > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > Houston, TX 77004
> > > > (512) 963-3262
> > >
> > >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/7799619f/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Sep  1 17:29:33 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 1 Sep 2015 17:29:33 -0500
Subject: [petsc-users] Integrating TAO into SNES and TS
In-Reply-To: <CAP2=TMhAipXm20+Ph5Qd=NT6JRp9Vb8AC2DVojXzh4uu1ynhEQ@mail.gmail.com>
References: <CAP2=TMgF8fKyX0o--rRz+cw01=d5ws6iCZu=2PVq1qBJq50okA@mail.gmail.com>
	<16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov>
	<CAP2=TMhD41MWMf=n+Z7ZdDdNYvRVx91fL956AXSjtJb_vz01gg@mail.gmail.com>
	<0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov>
	<CAP2=TMh=NSnObLKwgR_ZJHZDqNe_c+VO4NPkSFnA3X+NLd0Amg@mail.gmail.com>
	<6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov>
	<CAP2=TMiZXce6FBKMRNBtnyn3Ug7cnMn_2ZhgR2547=OqpO8Nog@mail.gmail.com>
	<C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
	<CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>
	<CAMYG4G=MTDKpRrfnV+nBZvCGWNhTe-qtSkmVoCOF4J_uLS=e4A@mail.gmail.com>
	<CAP2=TMjdp3iKJ8943dHwKQmFg1cb9WOt29MgEgaiNTmMMcmoow@mail.gmail.com>
	<CAMYG4GnvWpxNExr_VzkVt4q5UAEwMcHXMSYyahP9gxpP+-XRbw@mail.gmail.com>
	<CAP2=TMiJUz6yM5vMjThMTgZ6CmRsS+da6dU+4FuJq8qk6c+fdw@mail.gmail.com>
	<AB01DEC6-517D-48EE-AB00-504E42E9D5F1@mcs.anl.gov>
	<CAP2=TMhAipXm20+Ph5Qd=NT6JRp9Vb8AC2DVojXzh4uu1ynhEQ@mail.gmail.com>
Message-ID: <A41592D3-E2F7-4250-9BAD-6973377353C2@mcs.anl.gov>


  I really cannot understand what you are saying.

> On Sep 1, 2015, at 5:14 PM, Justin Chang <jychang48 at gmail.com> wrote:
> 
> So in VI, the i-th component/cell/point of x is "corrected" if it is below lb_i or above ub_i. All other x_i that satisfy their respective bounds remain untouched. Hence only the corrected components will violated the original equation

   I think what you say above is backwards. First of all x_i would never be < lb_i   or > ub_i since it is constrained to be between them. It is just that with a VI IF x_i is lb_i or ub_i then F_i(x) is allowed to be nonzero (that is that equation is allowed to violated).

> 
> In optimization, every single x_i will be "shifted" such that every single x_i meets lb_i and ub_i. Hence it's possible all components will violate the original equation

  No. With constraints in optimization, they must all be satisfied for one to say that one has a solution.

  With VIs the F(x) is NOT a constraint and should not be thought of as a constraint; only the bounds are constraints. Or think of F(x) as a special kind of constraint (that doesn't exist with optimization) that is allowed to be violated in a a very special way (when its x_i variable is on a constraint).

  Barry


> 
> Is that the general idea here?
> 
> On Tue, Sep 1, 2015 at 12:48 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>    I think we are talking past each other.
> 
>    With bound constraint VI's there is no optimization, there is an equation F(x) = 0 (where F may be linear or nonlinear) and constraints  a <= x <= c. With VIs the equation F_i(x) is simply not satisfied if x_i is on a bound (that is x_i = a_i or x_i = b_i),
> 
>    With optimization if you have an equality constraint and inequality constraints; if to satisfy an inequality constraint FORCES an equality constraint to not be satisfied then the constraints are not compatible and the problem isn't properly posed.
> 
>    Barry
> 
> 
> > On Sep 1, 2015, at 4:15 AM, Justin Chang <jychang48 at gmail.com> wrote:
> >
> > But if I add those linear equality constraint equations to my original problem, would they not be satisfied anyway? Say I add this to my weak form:
> >
> > Ax = b
> >
> > But once i subject x to some bounded constraints, Ax != b. Unless I add some sort of penalty where extra weighting is added to this property...
> >
> > On Tue, Sep 1, 2015 at 3:02 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > On Tue, Sep 1, 2015 at 3:46 AM, Justin Chang <jychang48 at gmail.com> wrote:
> > I would like to simultaneously enforce both discrete maximum principle and local mass/species balance. Because even if a locally conservative scheme like RT0 is used, as soon as these bounded constraints are applied, i lose the mass balance.
> >
> > What I am saying is, can't you just add "linear equality constraints" as more equations?
> >
> >   Thanks,
> >
> >     Matt
> >
> > On Tue, Sep 1, 2015 at 2:33 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > On Tue, Sep 1, 2015 at 3:11 AM, Justin Chang <jychang48 at gmail.com> wrote:
> > Barry,
> >
> > That's good to know thanks.
> >
> > On a related note, is it possible for VI to one day include linear equality constraints?
> >
> > How are these different from just using more equations?
> >
> >   Thanks,
> >
> >     Matt
> >
> > Thanks,
> > Justin
> >
> > On Mon, Aug 31, 2015 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Aug 31, 2015, at 7:36 PM, Justin Chang <jychang48 at gmail.com> wrote:
> > >
> > > Coming back to this,
> > >
> > > Say I now want to ensure the DMP for advection-diffusion equations. The linear operator is now asymmetric and non-self-adjoint (assuming I do something like SUPG or finite volume), meaning I cannot simply solve this problem without any manipulation (e.g. normalizing the equations) using TAO's optimization solvers. Does this statement also hold true for SNESVI?
> >
> >   SNESVI doesn't care about symmetry etc
> >
> > >
> > > Thanks,
> > > Justin
> > >
> > > On Fri, Apr 3, 2015 at 7:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >
> > > > On Apr 3, 2015, at 7:35 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > >
> > > > I guess I will have to write my own code then :)
> > > >
> > > > I am not all that familiar with Variational Inequalities at the moment, but if my Jacobian is symmetric and positive definite and I only have lower and upper bounds, doesn't the problem simply reduce to that of a convex optimization? That is, with SNES act as if it were Tao?
> > >
> > >   Yes, I think that is essentially correctly.
> > >
> > >   Barry
> > >
> > > >
> > > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > >
> > > >   Justin,
> > > >
> > > >    We haven't done anything with TS to handle variational inequalities. So you can either write your own backward Euler (outside of TS) that solves each time-step problem either as 1) an optimization problem using Tao or 2) as a variational inequality using SNES.
> > > >
> > > >    More adventurously you could look at the TSTHETA code in TS (which is a general form that includes Euler, Backward Euler and Crank-Nicolson and see if you can add the constraints to the SNES problem that is solved; in theory this is straightforward but it would require understanding the current code (which Jed, of course, overwrote :-). I think you should do this.
> > > >
> > > >   Barry
> > > >
> > > >
> > > > > On Apr 3, 2015, at 12:31 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > > >
> > > > > I am solving the following anisotropic transient diffusion equation subject to 0 bounds:
> > > > >
> > > > > du/dt = div[D*grad[u]] + f
> > > > >
> > > > > Where the dispersion tensor D(x) is symmetric and positive definite. This formulation violates the discrete maximum principles so one of the ways to ensure nonnegative concentrations is to employ convex optimization. I am following the procedures in Nakshatrala and Valocchi (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF.
> > > > >
> > > > > The Variational Inequality method works gives what I want for my transient case, but what if I want to implement the Tao methodology in TS? That is, what TS functions do I need to set up steps a) through e) for each time step (also the Jacobian remains the same for all time steps so I would only call this once). Normally I would just call TSSolve() and let the libraries and functions do everything, but I would like to incorporate TaoSolve into every time step.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > --
> > > > > Justin Chang
> > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > University of Houston, Department of Civil and Environmental Engineering
> > > > > Houston, TX 77004
> > > > > (512) 963-3262
> > > > >
> > > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > > >
> > > > >   An alternative approach is for you to solve it as a (non)linear variational inequality. See src/snes/examples/tutorials/ex9.c
> > > > >
> > > > >   How you should proceed depends on your long term goal. What problem do you really want to solve? Is it really a linear time dependent problem with 0 bounds on U? Can the problem always be represented as an optimization problem easily? What are  and what will be the properties of K? For example if K is positive definite then likely the bounds will remain try without explicitly providing the constraints.
> > > > >
> > > > >   Barry
> > > > >
> > > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I have a two part question regarding the integration of the following optimization problem
> > > > > >
> > > > > > min 1/2 u^T*K*u + u^T*f
> > > > > > S.T. u >= 0
> > > > > >
> > > > > > into SNES and TS
> > > > > >
> > > > > > 1) For SNES, assuming I am working with a linear FE equation, I have the following algorithm/steps for solving my problem
> > > > > >
> > > > > > a) Set an initial guess x
> > > > > > b) Obtain residual r and jacobian A through functions SNESComputeFunction() and SNESComputeJacobian() respectively
> > > > > > c) Form vector b = r - A*x
> > > > > > d) Set Hessian equal to A, gradient to A*x, objective function value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector
> > > > > > e) Call TaoSolve
> > > > > >
> > > > > > This works well at the moment, but my question is there a more "efficient" way of doing this? Because with my current setup, I am making a rather bold assumption that my problem would converge in one SNES iteration without the bounded constraints and does not have any unexpected nonlinearities.
> > > > > >
> > > > > > 2) How would I go about doing the above for time-stepping problems? At each time step, I want to solve a convex optimization subject to the lower bounds constraint. I plan on using backward euler and my resulting jacobian should still be compatible with the above optimization problem.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > --
> > > > > > Justin Chang
> > > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > > University of Houston, Department of Civil and Environmental Engineering
> > > > > > Houston, TX 77004
> > > > > > (512) 963-3262
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Justin Chang
> > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > University of Houston, Department of Civil and Environmental Engineering
> > > > > Houston, TX 77004
> > > > > (512) 963-3262
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Justin Chang
> > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > University of Houston, Department of Civil and Environmental Engineering
> > > > Houston, TX 77004
> > > > (512) 963-3262
> > >
> > >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> >
> 
> 


From jychang48 at gmail.com  Tue Sep  1 18:10:52 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 1 Sep 2015 17:10:52 -0600
Subject: [petsc-users] Integrating TAO into SNES and TS
In-Reply-To: <A41592D3-E2F7-4250-9BAD-6973377353C2@mcs.anl.gov>
References: <CAP2=TMgF8fKyX0o--rRz+cw01=d5ws6iCZu=2PVq1qBJq50okA@mail.gmail.com>
	<16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov>
	<CAP2=TMhD41MWMf=n+Z7ZdDdNYvRVx91fL956AXSjtJb_vz01gg@mail.gmail.com>
	<0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov>
	<CAP2=TMh=NSnObLKwgR_ZJHZDqNe_c+VO4NPkSFnA3X+NLd0Amg@mail.gmail.com>
	<6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov>
	<CAP2=TMiZXce6FBKMRNBtnyn3Ug7cnMn_2ZhgR2547=OqpO8Nog@mail.gmail.com>
	<C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
	<CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>
	<CAMYG4G=MTDKpRrfnV+nBZvCGWNhTe-qtSkmVoCOF4J_uLS=e4A@mail.gmail.com>
	<CAP2=TMjdp3iKJ8943dHwKQmFg1cb9WOt29MgEgaiNTmMMcmoow@mail.gmail.com>
	<CAMYG4GnvWpxNExr_VzkVt4q5UAEwMcHXMSYyahP9gxpP+-XRbw@mail.gmail.com>
	<CAP2=TMiJUz6yM5vMjThMTgZ6CmRsS+da6dU+4FuJq8qk6c+fdw@mail.gmail.com>
	<AB01DEC6-517D-48EE-AB00-504E42E9D5F1@mcs.anl.gov>
	<CAP2=TMhAipXm20+Ph5Qd=NT6JRp9Vb8AC2DVojXzh4uu1ynhEQ@mail.gmail.com>
	<A41592D3-E2F7-4250-9BAD-6973377353C2@mcs.anl.gov>
Message-ID: <CAP2=TMiHGnQRCXJJ7Ry1tAb0Z-EovE_w2ZTY40qNCzqQ6uEc6A@mail.gmail.com>

Let me back up a bit.

Say I am solving -div[D(x)grad[c]] = 0 on a unit cube with a hole. Interior
boundary has DirichletBC c = 1 and exterior boundary  has DirichletBC c =
0. If I solve the above in the strong sense (e.g., finite volume, finite
difference, etc) -div[D(x)grad[c]] = 0 at every grid cell (hence local mass
conservation).

According to discrete maximum principle and the prescribed BCs, my bounds
*should* be between 0 <= c <= 1. However, if D(x) is aniosotropic, the
KSP/SNES solver will return c's less than 0 and/or greater than 1. Local
mass balance still exists for all cells regardless.

Now, if TAO is used and Vec XL and XU of TaoSetVariable bounds are set to 0
and 1 respectively, I now ensure that 0 <= c <= 1 even for anisotropy.
However, our experiments have shown that the "corrected" regions i.e., the
regions where it would have violated DMP otherwise, no longer satisfy local
mass balance i.e., div[D(x)grad[c]] != at the element level. The unaffected
regions, i.e. the regions where DMP was satisfied regardless, are still
locally conservative.

We have shown through MATLAB's *quadprog* that we can achieve both DMP and
local mass balance if we applied the mixed form (using least-squares finite
element method) of the diffusion and subject the optimization solve to
satisfy both upper/lower bounds as well as equality constraints (which is
divergence of flux = 0 in Omega). If no bounds/constraints were set, the
objective function is equal to zero. If either only bounds or constraints,
the objective function is equal to some non-zero value. If both bounds and
constraints, the objective function is a slightly larger non-zero value.
Basically, the more constraints I apply, the further and further away my
overall solution may drift from the original discretization.

I am curious to see if 1) something like this is possible with TAO and 2)
if I can achieve the SNESVI equivalent in the sense I have described. Hope
this makes a little more sense.

Thanks,
Justin

On Tue, Sep 1, 2015 at 4:29 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   I really cannot understand what you are saying.
>
> > On Sep 1, 2015, at 5:14 PM, Justin Chang <jychang48 at gmail.com> wrote:
> >
> > So in VI, the i-th component/cell/point of x is "corrected" if it is
> below lb_i or above ub_i. All other x_i that satisfy their respective
> bounds remain untouched. Hence only the corrected components will violated
> the original equation
>
>    I think what you say above is backwards. First of all x_i would never
> be < lb_i   or > ub_i since it is constrained to be between them. It is
> just that with a VI IF x_i is lb_i or ub_i then F_i(x) is allowed to be
> nonzero (that is that equation is allowed to violated).
>
> >
> > In optimization, every single x_i will be "shifted" such that every
> single x_i meets lb_i and ub_i. Hence it's possible all components will
> violate the original equation
>
>   No. With constraints in optimization, they must all be satisfied for one
> to say that one has a solution.
>
>   With VIs the F(x) is NOT a constraint and should not be thought of as a
> constraint; only the bounds are constraints. Or think of F(x) as a special
> kind of constraint (that doesn't exist with optimization) that is allowed
> to be violated in a a very special way (when its x_i variable is on a
> constraint).
>
>   Barry
>
>
> >
> > Is that the general idea here?
> >
> > On Tue, Sep 1, 2015 at 12:48 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >    I think we are talking past each other.
> >
> >    With bound constraint VI's there is no optimization, there is an
> equation F(x) = 0 (where F may be linear or nonlinear) and constraints  a
> <= x <= c. With VIs the equation F_i(x) is simply not satisfied if x_i is
> on a bound (that is x_i = a_i or x_i = b_i),
> >
> >    With optimization if you have an equality constraint and inequality
> constraints; if to satisfy an inequality constraint FORCES an equality
> constraint to not be satisfied then the constraints are not compatible and
> the problem isn't properly posed.
> >
> >    Barry
> >
> >
> > > On Sep 1, 2015, at 4:15 AM, Justin Chang <jychang48 at gmail.com> wrote:
> > >
> > > But if I add those linear equality constraint equations to my original
> problem, would they not be satisfied anyway? Say I add this to my weak form:
> > >
> > > Ax = b
> > >
> > > But once i subject x to some bounded constraints, Ax != b. Unless I
> add some sort of penalty where extra weighting is added to this property...
> > >
> > > On Tue, Sep 1, 2015 at 3:02 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > > On Tue, Sep 1, 2015 at 3:46 AM, Justin Chang <jychang48 at gmail.com>
> wrote:
> > > I would like to simultaneously enforce both discrete maximum principle
> and local mass/species balance. Because even if a locally conservative
> scheme like RT0 is used, as soon as these bounded constraints are applied,
> i lose the mass balance.
> > >
> > > What I am saying is, can't you just add "linear equality constraints"
> as more equations?
> > >
> > >   Thanks,
> > >
> > >     Matt
> > >
> > > On Tue, Sep 1, 2015 at 2:33 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > > On Tue, Sep 1, 2015 at 3:11 AM, Justin Chang <jychang48 at gmail.com>
> wrote:
> > > Barry,
> > >
> > > That's good to know thanks.
> > >
> > > On a related note, is it possible for VI to one day include linear
> equality constraints?
> > >
> > > How are these different from just using more equations?
> > >
> > >   Thanks,
> > >
> > >     Matt
> > >
> > > Thanks,
> > > Justin
> > >
> > > On Mon, Aug 31, 2015 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > > > On Aug 31, 2015, at 7:36 PM, Justin Chang <jychang48 at gmail.com>
> wrote:
> > > >
> > > > Coming back to this,
> > > >
> > > > Say I now want to ensure the DMP for advection-diffusion equations.
> The linear operator is now asymmetric and non-self-adjoint (assuming I do
> something like SUPG or finite volume), meaning I cannot simply solve this
> problem without any manipulation (e.g. normalizing the equations) using
> TAO's optimization solvers. Does this statement also hold true for SNESVI?
> > >
> > >   SNESVI doesn't care about symmetry etc
> > >
> > > >
> > > > Thanks,
> > > > Justin
> > > >
> > > > On Fri, Apr 3, 2015 at 7:38 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > >
> > > > > On Apr 3, 2015, at 7:35 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > > >
> > > > > I guess I will have to write my own code then :)
> > > > >
> > > > > I am not all that familiar with Variational Inequalities at the
> moment, but if my Jacobian is symmetric and positive definite and I only
> have lower and upper bounds, doesn't the problem simply reduce to that of a
> convex optimization? That is, with SNES act as if it were Tao?
> > > >
> > > >   Yes, I think that is essentially correctly.
> > > >
> > > >   Barry
> > > >
> > > > >
> > > > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > >
> > > > >   Justin,
> > > > >
> > > > >    We haven't done anything with TS to handle variational
> inequalities. So you can either write your own backward Euler (outside of
> TS) that solves each time-step problem either as 1) an optimization problem
> using Tao or 2) as a variational inequality using SNES.
> > > > >
> > > > >    More adventurously you could look at the TSTHETA code in TS
> (which is a general form that includes Euler, Backward Euler and
> Crank-Nicolson and see if you can add the constraints to the SNES problem
> that is solved; in theory this is straightforward but it would require
> understanding the current code (which Jed, of course, overwrote :-). I
> think you should do this.
> > > > >
> > > > >   Barry
> > > > >
> > > > >
> > > > > > On Apr 3, 2015, at 12:31 PM, Justin Chang <jchang27 at uh.edu>
> wrote:
> > > > > >
> > > > > > I am solving the following anisotropic transient diffusion
> equation subject to 0 bounds:
> > > > > >
> > > > > > du/dt = div[D*grad[u]] + f
> > > > > >
> > > > > > Where the dispersion tensor D(x) is symmetric and positive
> definite. This formulation violates the discrete maximum principles so one
> of the ways to ensure nonnegative concentrations is to employ convex
> optimization. I am following the procedures in Nakshatrala and Valocchi
> (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF.
> > > > > >
> > > > > > The Variational Inequality method works gives what I want for my
> transient case, but what if I want to implement the Tao methodology in TS?
> That is, what TS functions do I need to set up steps a) through e) for each
> time step (also the Jacobian remains the same for all time steps so I would
> only call this once). Normally I would just call TSSolve() and let the
> libraries and functions do everything, but I would like to incorporate
> TaoSolve into every time step.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > --
> > > > > > Justin Chang
> > > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > > > Houston, TX 77004
> > > > > > (512) 963-3262
> > > > > >
> > > > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > > >
> > > > > >   An alternative approach is for you to solve it as a
> (non)linear variational inequality. See src/snes/examples/tutorials/ex9.c
> > > > > >
> > > > > >   How you should proceed depends on your long term goal. What
> problem do you really want to solve? Is it really a linear time dependent
> problem with 0 bounds on U? Can the problem always be represented as an
> optimization problem easily? What are  and what will be the properties of
> K? For example if K is positive definite then likely the bounds will remain
> try without explicitly providing the constraints.
> > > > > >
> > > > > >   Barry
> > > > > >
> > > > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang <jchang27 at uh.edu>
> wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I have a two part question regarding the integration of the
> following optimization problem
> > > > > > >
> > > > > > > min 1/2 u^T*K*u + u^T*f
> > > > > > > S.T. u >= 0
> > > > > > >
> > > > > > > into SNES and TS
> > > > > > >
> > > > > > > 1) For SNES, assuming I am working with a linear FE equation,
> I have the following algorithm/steps for solving my problem
> > > > > > >
> > > > > > > a) Set an initial guess x
> > > > > > > b) Obtain residual r and jacobian A through functions
> SNESComputeFunction() and SNESComputeJacobian() respectively
> > > > > > > c) Form vector b = r - A*x
> > > > > > > d) Set Hessian equal to A, gradient to A*x, objective function
> value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector
> > > > > > > e) Call TaoSolve
> > > > > > >
> > > > > > > This works well at the moment, but my question is there a more
> "efficient" way of doing this? Because with my current setup, I am making a
> rather bold assumption that my problem would converge in one SNES iteration
> without the bounded constraints and does not have any unexpected
> nonlinearities.
> > > > > > >
> > > > > > > 2) How would I go about doing the above for time-stepping
> problems? At each time step, I want to solve a convex optimization subject
> to the lower bounds constraint. I plan on using backward euler and my
> resulting jacobian should still be compatible with the above optimization
> problem.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > --
> > > > > > > Justin Chang
> > > > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > > > > Houston, TX 77004
> > > > > > > (512) 963-3262
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Justin Chang
> > > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > > > Houston, TX 77004
> > > > > > (512) 963-3262
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Justin Chang
> > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > University of Houston, Department of Civil and Environmental
> Engineering
> > > > > Houston, TX 77004
> > > > > (512) 963-3262
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > -- Norbert Wiener
> > >
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > -- Norbert Wiener
> > >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150901/1e314490/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Sep  1 18:26:07 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 1 Sep 2015 18:26:07 -0500
Subject: [petsc-users] Integrating TAO into SNES and TS
In-Reply-To: <CAP2=TMiHGnQRCXJJ7Ry1tAb0Z-EovE_w2ZTY40qNCzqQ6uEc6A@mail.gmail.com>
References: <CAP2=TMgF8fKyX0o--rRz+cw01=d5ws6iCZu=2PVq1qBJq50okA@mail.gmail.com>
	<16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov>
	<CAP2=TMhD41MWMf=n+Z7ZdDdNYvRVx91fL956AXSjtJb_vz01gg@mail.gmail.com>
	<0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov>
	<CAP2=TMh=NSnObLKwgR_ZJHZDqNe_c+VO4NPkSFnA3X+NLd0Amg@mail.gmail.com>
	<6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov>
	<CAP2=TMiZXce6FBKMRNBtnyn3Ug7cnMn_2ZhgR2547=OqpO8Nog@mail.gmail.com>
	<C9CB7856-391E-42A0-9EE7-C2D4C1A7E1A7@mcs.anl.gov>
	<CAP2=TMgEnYxacoh__WOQUvCC=rgvNcZ2MwEi6YHeDZBcY8-YvQ@mail.gmail.com>
	<CAMYG4G=MTDKpRrfnV+nBZvCGWNhTe-qtSkmVoCOF4J_uLS=e4A@mail.gmail.com>
	<CAP2=TMjdp3iKJ8943dHwKQmFg1cb9WOt29MgEgaiNTmMMcmoow@mail.gmail.com>
	<CAMYG4GnvWpxNExr_VzkVt4q5UAEwMcHXMSYyahP9gxpP+-XRbw@mail.gmail.com>
	<CAP2=TMiJUz6yM5vMjThMTgZ6CmRsS+da6dU+4FuJq8qk6c+fdw@mail.gmail.com>
	<AB01DEC6-517D-48EE-AB00-504E42E9D5F1@mcs.anl.gov>
	<CAP2=TMhAipXm20+Ph5Qd=NT6JRp9Vb8AC2DVojXzh4uu1ynhEQ@mail.gmail.com>
	<A41592D3-E2F7-4250-9BAD-6973377353C2@mcs.anl.gov>
	<CAP2=TMiHGnQRCXJJ7Ry1tAb0Z-EovE_w2ZTY40qNCzqQ6uEc6A@mail.gmail.com>
Message-ID: <BE327C85-AC6A-4A10-9F67-0251715E26C0@mcs.anl.gov>


> On Sep 1, 2015, at 6:10 PM, Justin Chang <jychang48 at gmail.com> wrote:
> 
> Let me back up a bit.
> 
> Say I am solving -div[D(x)grad[c]] = 0 on a unit cube with a hole. Interior boundary has DirichletBC c = 1 and exterior boundary  has DirichletBC c = 0. If I solve the above in the strong sense (e.g., finite volume, finite difference, etc) -div[D(x)grad[c]] = 0 at every grid cell (hence local mass conservation).
> 
> According to discrete maximum principle and the prescribed BCs, my bounds should be between 0 <= c <= 1. However, if D(x) is aniosotropic, the KSP/SNES solver will return c's less than 0 and/or greater than 1. Local mass balance still exists for all cells regardless.
> 
> Now, if TAO is used and Vec XL and XU of TaoSetVariable bounds are set to 0 and 1 respectively, I now ensure that 0 <= c <= 1 even for anisotropy. However, our experiments have shown that the "corrected" regions i.e., the regions where it would have violated DMP otherwise, no longer satisfy local mass balance i.e., div[D(x)grad[c]] != at the element level. The unaffected regions, i.e. the regions where DMP was satisfied regardless, are still locally conservative.
> 
> We have shown through MATLAB's quadprog that we can achieve both DMP and local mass balance if we applied the mixed form (using least-squares finite element method) of the diffusion and subject the optimization solve to satisfy both upper/lower bounds as well as equality constraints (which is divergence of flux = 0 in Omega). If no bounds/constraints were set, the objective function is equal to zero. If either only bounds or constraints, the objective function is equal to some non-zero value. If both bounds and constraints, the objective function is a slightly larger non-zero value. Basically, the more constraints I apply, the further and further away my overall solution may drift from the original discretization.
> 
> I am curious to see if 1) something like this is possible with TAO

   I don't really understand what you want to do, but if it is an optimization problem with constraints, at least in theory it can do it.

> and 2) if I can achieve the SNESVI equivalent in the sense I have described.

   I doubt it, because the very definition of VIs says that at inequality constraints the equation is not satisfied (and it cannot be because there is no solution that satisfies the equations AND satisfies the bounds).

  Barry

   
> Hope this makes a little more sense.
> 
> Thanks,
> Justin
> 
> On Tue, Sep 1, 2015 at 4:29 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   I really cannot understand what you are saying.
> 
> > On Sep 1, 2015, at 5:14 PM, Justin Chang <jychang48 at gmail.com> wrote:
> >
> > So in VI, the i-th component/cell/point of x is "corrected" if it is below lb_i or above ub_i. All other x_i that satisfy their respective bounds remain untouched. Hence only the corrected components will violated the original equation
> 
>    I think what you say above is backwards. First of all x_i would never be < lb_i   or > ub_i since it is constrained to be between them. It is just that with a VI IF x_i is lb_i or ub_i then F_i(x) is allowed to be nonzero (that is that equation is allowed to violated).
> 
> >
> > In optimization, every single x_i will be "shifted" such that every single x_i meets lb_i and ub_i. Hence it's possible all components will violate the original equation
> 
>   No. With constraints in optimization, they must all be satisfied for one to say that one has a solution.
> 
>   With VIs the F(x) is NOT a constraint and should not be thought of as a constraint; only the bounds are constraints. Or think of F(x) as a special kind of constraint (that doesn't exist with optimization) that is allowed to be violated in a a very special way (when its x_i variable is on a constraint).
> 
>   Barry
> 
> 
> >
> > Is that the general idea here?
> >
> > On Tue, Sep 1, 2015 at 12:48 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >    I think we are talking past each other.
> >
> >    With bound constraint VI's there is no optimization, there is an equation F(x) = 0 (where F may be linear or nonlinear) and constraints  a <= x <= c. With VIs the equation F_i(x) is simply not satisfied if x_i is on a bound (that is x_i = a_i or x_i = b_i),
> >
> >    With optimization if you have an equality constraint and inequality constraints; if to satisfy an inequality constraint FORCES an equality constraint to not be satisfied then the constraints are not compatible and the problem isn't properly posed.
> >
> >    Barry
> >
> >
> > > On Sep 1, 2015, at 4:15 AM, Justin Chang <jychang48 at gmail.com> wrote:
> > >
> > > But if I add those linear equality constraint equations to my original problem, would they not be satisfied anyway? Say I add this to my weak form:
> > >
> > > Ax = b
> > >
> > > But once i subject x to some bounded constraints, Ax != b. Unless I add some sort of penalty where extra weighting is added to this property...
> > >
> > > On Tue, Sep 1, 2015 at 3:02 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > > On Tue, Sep 1, 2015 at 3:46 AM, Justin Chang <jychang48 at gmail.com> wrote:
> > > I would like to simultaneously enforce both discrete maximum principle and local mass/species balance. Because even if a locally conservative scheme like RT0 is used, as soon as these bounded constraints are applied, i lose the mass balance.
> > >
> > > What I am saying is, can't you just add "linear equality constraints" as more equations?
> > >
> > >   Thanks,
> > >
> > >     Matt
> > >
> > > On Tue, Sep 1, 2015 at 2:33 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > > On Tue, Sep 1, 2015 at 3:11 AM, Justin Chang <jychang48 at gmail.com> wrote:
> > > Barry,
> > >
> > > That's good to know thanks.
> > >
> > > On a related note, is it possible for VI to one day include linear equality constraints?
> > >
> > > How are these different from just using more equations?
> > >
> > >   Thanks,
> > >
> > >     Matt
> > >
> > > Thanks,
> > > Justin
> > >
> > > On Mon, Aug 31, 2015 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >
> > > > On Aug 31, 2015, at 7:36 PM, Justin Chang <jychang48 at gmail.com> wrote:
> > > >
> > > > Coming back to this,
> > > >
> > > > Say I now want to ensure the DMP for advection-diffusion equations. The linear operator is now asymmetric and non-self-adjoint (assuming I do something like SUPG or finite volume), meaning I cannot simply solve this problem without any manipulation (e.g. normalizing the equations) using TAO's optimization solvers. Does this statement also hold true for SNESVI?
> > >
> > >   SNESVI doesn't care about symmetry etc
> > >
> > > >
> > > > Thanks,
> > > > Justin
> > > >
> > > > On Fri, Apr 3, 2015 at 7:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > >
> > > > > On Apr 3, 2015, at 7:35 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > > >
> > > > > I guess I will have to write my own code then :)
> > > > >
> > > > > I am not all that familiar with Variational Inequalities at the moment, but if my Jacobian is symmetric and positive definite and I only have lower and upper bounds, doesn't the problem simply reduce to that of a convex optimization? That is, with SNES act as if it were Tao?
> > > >
> > > >   Yes, I think that is essentially correctly.
> > > >
> > > >   Barry
> > > >
> > > > >
> > > > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > > >
> > > > >   Justin,
> > > > >
> > > > >    We haven't done anything with TS to handle variational inequalities. So you can either write your own backward Euler (outside of TS) that solves each time-step problem either as 1) an optimization problem using Tao or 2) as a variational inequality using SNES.
> > > > >
> > > > >    More adventurously you could look at the TSTHETA code in TS (which is a general form that includes Euler, Backward Euler and Crank-Nicolson and see if you can add the constraints to the SNES problem that is solved; in theory this is straightforward but it would require understanding the current code (which Jed, of course, overwrote :-). I think you should do this.
> > > > >
> > > > >   Barry
> > > > >
> > > > >
> > > > > > On Apr 3, 2015, at 12:31 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > > > >
> > > > > > I am solving the following anisotropic transient diffusion equation subject to 0 bounds:
> > > > > >
> > > > > > du/dt = div[D*grad[u]] + f
> > > > > >
> > > > > > Where the dispersion tensor D(x) is symmetric and positive definite. This formulation violates the discrete maximum principles so one of the ways to ensure nonnegative concentrations is to employ convex optimization. I am following the procedures in Nakshatrala and Valocchi (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF.
> > > > > >
> > > > > > The Variational Inequality method works gives what I want for my transient case, but what if I want to implement the Tao methodology in TS? That is, what TS functions do I need to set up steps a) through e) for each time step (also the Jacobian remains the same for all time steps so I would only call this once). Normally I would just call TSSolve() and let the libraries and functions do everything, but I would like to incorporate TaoSolve into every time step.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > --
> > > > > > Justin Chang
> > > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > > University of Houston, Department of Civil and Environmental Engineering
> > > > > > Houston, TX 77004
> > > > > > (512) 963-3262
> > > > > >
> > > > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > > > >
> > > > > >   An alternative approach is for you to solve it as a (non)linear variational inequality. See src/snes/examples/tutorials/ex9.c
> > > > > >
> > > > > >   How you should proceed depends on your long term goal. What problem do you really want to solve? Is it really a linear time dependent problem with 0 bounds on U? Can the problem always be represented as an optimization problem easily? What are  and what will be the properties of K? For example if K is positive definite then likely the bounds will remain try without explicitly providing the constraints.
> > > > > >
> > > > > >   Barry
> > > > > >
> > > > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang <jchang27 at uh.edu> wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I have a two part question regarding the integration of the following optimization problem
> > > > > > >
> > > > > > > min 1/2 u^T*K*u + u^T*f
> > > > > > > S.T. u >= 0
> > > > > > >
> > > > > > > into SNES and TS
> > > > > > >
> > > > > > > 1) For SNES, assuming I am working with a linear FE equation, I have the following algorithm/steps for solving my problem
> > > > > > >
> > > > > > > a) Set an initial guess x
> > > > > > > b) Obtain residual r and jacobian A through functions SNESComputeFunction() and SNESComputeJacobian() respectively
> > > > > > > c) Form vector b = r - A*x
> > > > > > > d) Set Hessian equal to A, gradient to A*x, objective function value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector
> > > > > > > e) Call TaoSolve
> > > > > > >
> > > > > > > This works well at the moment, but my question is there a more "efficient" way of doing this? Because with my current setup, I am making a rather bold assumption that my problem would converge in one SNES iteration without the bounded constraints and does not have any unexpected nonlinearities.
> > > > > > >
> > > > > > > 2) How would I go about doing the above for time-stepping problems? At each time step, I want to solve a convex optimization subject to the lower bounds constraint. I plan on using backward euler and my resulting jacobian should still be compatible with the above optimization problem.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > --
> > > > > > > Justin Chang
> > > > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > > > University of Houston, Department of Civil and Environmental Engineering
> > > > > > > Houston, TX 77004
> > > > > > > (512) 963-3262
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Justin Chang
> > > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > > University of Houston, Department of Civil and Environmental Engineering
> > > > > > Houston, TX 77004
> > > > > > (512) 963-3262
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Justin Chang
> > > > > PhD Candidate, Civil Engineering - Computational Sciences
> > > > > University of Houston, Department of Civil and Environmental Engineering
> > > > > Houston, TX 77004
> > > > > (512) 963-3262
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > > -- Norbert Wiener
> > >
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > > -- Norbert Wiener
> > >
> >
> >
> 
> 


From filippo.leon at gmail.com  Wed Sep  2 01:52:21 2015
From: filippo.leon at gmail.com (Filippo Leonardi)
Date: Wed, 02 Sep 2015 08:52:21 +0200
Subject: [petsc-users] 2D Poisson on nonuniform meshes
In-Reply-To: <CAMYG4GkVnj4tJD5gdeUiOLxgKOUhDkg16nM6MpB7yRc3QeSSAw@mail.gmail.com>
References: <3634751.Hfn5OcL7V3@besikowitch-iii>
	<2124074.mjk2bZhIH0@besikowitch-iii>
	<CAMYG4GkVnj4tJD5gdeUiOLxgKOUhDkg16nM6MpB7yRc3QeSSAw@mail.gmail.com>
Message-ID: <1970955.AlfxvCWIyV@besikowitch-iii>

So I spent a lot of time debugging my code for this one, it turns out that in my debug run, where I 
solve the same system multiple times, reusing the preconditioner if possible, sometimes, it 
happens that the rhs is 0. It has nothing to to with scaling. Just that when I did my debug run I 
just had reworked the code to use nonuniform meshes, so I assumed incorrectly, hey, there is an 
issue with scaling.

I do not perform the checks for incompatible rhs in this case, despite knowing that PETSc breaks 
in this case (I knew that, but I forgot). I now introduced a check in my debug build that check if a 
rhs is zero or not.

Long story short:
I have the suspect that PETSc could be improved in this respect: I am not an expert but 
somehow it shouldn't break or should warn the user (even if he is doing somthing "nonsensical" 
like solving Ax = 0). Could it be that if I set PETSc to solve a system with rhs 0 it breaks (maybe 
if combined with a multiple solve?)? What I do is the following:

i approximate the Leray projection operator with Laplace, so i solve Ax = b multiple times, 
reusing the matrix/precond. When I debug the code, sometimes I have a b = 0.

(In fact, this was my first misktake:
http://lists.mcs.anl.gov/pipermail/petsc-users/2014-November/023478.html
)

Maybe because the relative error cannot possibly work? If so, can I suggest to insert some sort 
of check for idiots like me :) (I repeated this mistake twice knowing what the issue was)?

On Tuesday 01 September 2015 12:13:34 Matthew Knepley wrote:
> On Tue, Sep 1, 2015 at 6:53 AM, Filippo Leonardi <filippo.leon at gmail.com>
> 
> wrote:
> > On Tuesday 01 September 2015 06:32:38 you wrote:
> > > > > 1) The KSP view does not say it is shifting. Are you using the
> > > > > latest
> > > > > 
> > > > > 
> > > > > 
> > > > > release?
> > > > 
> > > > yes, 3.6. Does PETSc warn for that even if I set the nullspace? I can
> > 
> > also
> > 
> > > > check MUMPS or something else.
> > > 
> > > I am not sure what you think PETSc does here. If shifting were enabled,
> > 
> > it
> > 
> > > would add some
> > > 
> > > diagonal matrix to the input matrix and continue the factorization. This
> > > 
> > > would mean that the
> > > 
> > > factors were not, in fact, a factorization of the input matrix, and you
> > > 
> > > would not get the exact
> > > 
> > > solution in one iterate.
> > 
> > I though PETSc would've replace my pivots with small eps, which is
> > actually not a problem in my case
> > 
> > > > > 2) If it shifted, it would not solve in a single iterate.
> > > > 
> > > > even with preonly?
> > > 
> > > You would have a large residual. Do you?
> > 
> > Actually, I get a perfect solution.
> > 
> > > > > 3) Your GAMG results imply that something is wrong with the coarse
> > > > > 
> > > > > solve.
> > > > > 
> > > > > 
> > > > > 
> > > > > This is exactly what would happen if
> > > > > 
> > > > > 
> > > > > 
> > > > > that problem was not solved accurately (its off by > 10 orders of
> > > > > 
> > > > > 
> > > > > 
> > > > > magnitude).
> > > > 
> > > > yes, but GAMG builds is own coarse solvers so either the problem is
> > > > 
> > > > already in the definition of A and b (likely) or it is a bug in gamg.
> > > 
> > > Yes. GAMG uses the constants to build the basis, on the assumption that
> > > 
> > > they are in the (near) nullspace of the
> > > 
> > > operator with no boundary conditions. Since this is far off, I think
> > > this
> > > 
> > > must not be true for your A.
> > > 
> > > > > It sounds like your operator is not singular, and its not the
> > 
> > Laplacian
> > 
> > > > > since it does not look like the Neumann version
> > > > > 
> > > > > 
> > > > > 
> > > > > has constants as a null space.
> > > > 
> > > > I'm using periodic boundaries, and constants are in kern(A)
> > > 
> > > Did you check?
> > 
> > Checked with VecSet and MatMult just in case, I get a machine eps constant
> > vector.
> 
> Okay, it seems that we have the best chance of figuring out the problem by
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150902/fb030d61/attachment-0001.html>

From bsmith at mcs.anl.gov  Wed Sep  2 12:42:44 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 2 Sep 2015 12:42:44 -0500
Subject: [petsc-users] 2D Poisson on nonuniform meshes
In-Reply-To: <1970955.AlfxvCWIyV@besikowitch-iii>
References: <3634751.Hfn5OcL7V3@besikowitch-iii>
	<2124074.mjk2bZhIH0@besikowitch-iii>
	<CAMYG4GkVnj4tJD5gdeUiOLxgKOUhDkg16nM6MpB7yRc3QeSSAw@mail.gmail.com>
	<1970955.AlfxvCWIyV@besikowitch-iii>
Message-ID: <A97CA9E8-5487-485C-8168-73F712EAA383@mcs.anl.gov>


  We are changing the code to always use a random right hand side for computing the Chebyshev estimates instead of the given right hand side so hopefully this problem will go away in the near future.

   Barry

> On Sep 2, 2015, at 1:52 AM, Filippo Leonardi <filippo.leon at gmail.com> wrote:
> 
> So I spent a lot of time debugging my code for this one, it turns out that in my debug run, where I solve the same system multiple times, reusing the preconditioner if possible, sometimes, it happens that the rhs is 0. It has nothing to to with scaling. Just that when I did my debug run I just had reworked the code to use nonuniform meshes, so I assumed incorrectly, hey, there is an issue with scaling.
>  
> I do not perform the checks for incompatible rhs in this case, despite knowing that PETSc breaks in this case (I knew that, but I forgot). I now introduced a check in my debug build that check if a rhs is zero or not.
>  
> Long story short:
> I have the suspect that PETSc could be improved in this respect: I am not an expert but somehow it shouldn't break or should warn the user (even if he is doing somthing "nonsensical" like solving Ax = 0). Could it be that if I set PETSc to solve a system with rhs 0 it breaks (maybe if combined with a multiple solve?)? What I do is the following:
>  
> i approximate the Leray projection operator with Laplace, so i solve Ax = b multiple times, reusing the matrix/precond. When I debug the code, sometimes I have a b = 0.
>  
> (In fact, this was my first misktake:
> http://lists.mcs.anl.gov/pipermail/petsc-users/2014-November/023478.html
> )
>  
> Maybe because the relative error cannot possibly work? If so, can I suggest to insert some sort of check for idiots like me :) (I repeated this mistake twice knowing what the issue was)?
>  
> On Tuesday 01 September 2015 12:13:34 Matthew Knepley wrote:
> > On Tue, Sep 1, 2015 at 6:53 AM, Filippo Leonardi <filippo.leon at gmail.com>
> > 
> > wrote:
> > > On Tuesday 01 September 2015 06:32:38 you wrote:
> > > > > > 1) The KSP view does not say it is shifting. Are you using the
> > > > > > latest
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > release?
> > > > > 
> > > > > yes, 3.6. Does PETSc warn for that even if I set the nullspace? I can
> > > 
> > > also
> > > 
> > > > > check MUMPS or something else.
> > > > 
> > > > I am not sure what you think PETSc does here. If shifting were enabled,
> > > 
> > > it
> > > 
> > > > would add some
> > > > 
> > > > diagonal matrix to the input matrix and continue the factorization. This
> > > > 
> > > > would mean that the
> > > > 
> > > > factors were not, in fact, a factorization of the input matrix, and you
> > > > 
> > > > would not get the exact
> > > > 
> > > > solution in one iterate.
> > > 
> > > I though PETSc would've replace my pivots with small eps, which is
> > > actually not a problem in my case
> > > 
> > > > > > 2) If it shifted, it would not solve in a single iterate.
> > > > > 
> > > > > even with preonly?
> > > > 
> > > > You would have a large residual. Do you?
> > > 
> > > Actually, I get a perfect solution.
> > > 
> > > > > > 3) Your GAMG results imply that something is wrong with the coarse
> > > > > > 
> > > > > > solve.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > This is exactly what would happen if
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > that problem was not solved accurately (its off by > 10 orders of
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > magnitude).
> > > > > 
> > > > > yes, but GAMG builds is own coarse solvers so either the problem is
> > > > > 
> > > > > already in the definition of A and b (likely) or it is a bug in gamg.
> > > > 
> > > > Yes. GAMG uses the constants to build the basis, on the assumption that
> > > > 
> > > > they are in the (near) nullspace of the
> > > > 
> > > > operator with no boundary conditions. Since this is far off, I think
> > > > this
> > > > 
> > > > must not be true for your A.
> > > > 
> > > > > > It sounds like your operator is not singular, and its not the
> > > 
> > > Laplacian
> > > 
> > > > > > since it does not look like the Neumann version
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > has constants as a null space.
> > > > > 
> > > > > I'm using periodic boundaries, and constants are in kern(A)
> > > > 
> > > > Did you check?
> > > 
> > > Checked with VecSet and MatMult just in case, I get a machine eps constant
> > > vector.
> > 
> > Okay, it seems that we have the best chance of figuring out the problem by
> > looking at option 2) and 4).
> > 
> > First, is your rhs consistent, meaning is it orthogonal to your nullspace?
> > 
> >    Matt
> > 
> > > > Matt
> > > > 
> > > > > Thanks a lot for your time.
> > > > > 
> > > > > > Thanks,


From xzhao99 at gmail.com  Thu Sep  3 10:11:34 2015
From: xzhao99 at gmail.com (Xujun Zhao)
Date: Thu, 3 Sep 2015 10:11:34 -0500
Subject: [petsc-users] PETSc random number generator
Message-ID: <CAHOKZ65BOWN5NrusJkBqk8a+H5MFcXCSHf7dZY-U6EWd30-WSw@mail.gmail.com>

Hi all,

I need random Vectors with both Gaussian distribution and uniform
distribution. I notice that PETSc has VecSetRandom() function. I am curious
what types of distribution this function can generate. Thank you for your
answer.

Xujun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/5104c0ae/attachment.html>

From bsmith at mcs.anl.gov  Thu Sep  3 10:48:28 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 Sep 2015 10:48:28 -0500
Subject: [petsc-users] PETSc random number generator
In-Reply-To: <CAHOKZ65BOWN5NrusJkBqk8a+H5MFcXCSHf7dZY-U6EWd30-WSw@mail.gmail.com>
References: <CAHOKZ65BOWN5NrusJkBqk8a+H5MFcXCSHf7dZY-U6EWd30-WSw@mail.gmail.com>
Message-ID: <3E0FF19C-955B-485F-B7A8-EF48B86C11C8@mcs.anl.gov>


> On Sep 3, 2015, at 10:11 AM, Xujun Zhao <xzhao99 at gmail.com> wrote:
> 
> Hi all,
> 
> I need random Vectors with both Gaussian distribution and uniform distribution. I notice that PETSc has VecSetRandom() function. I am curious what types of distribution this function can generate. Thank you for your answer.

  Currently we have only uniform.

  I recommend using another library for the Gaussian and using VecGetArray() plus VecGetLocalSize() to loop over local entries putting in your random numbers.

   Barry

> 
> Xujun


From pvsang002 at gmail.com  Thu Sep  3 11:28:10 2015
From: pvsang002 at gmail.com (Sang pham van)
Date: Fri, 4 Sep 2015 00:28:10 +0800
Subject: [petsc-users] DMCreateMatrix with some dense row
Message-ID: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>

Hi,

I am using DMCreateMatrix to create matrix from a existed DM object and
defined stencil.
In my code, boundary nodes need to involve many inner nodes, thus matrix
rows corresponding to boundary nodes are almost dense. How can I  tell
petsc that those rows need to be preallocated with more entries? I don't
want to use MatMPIAIJSetPreallocation() since advantages of DM might be
lost.

Many thanks.

Sam.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150904/6fdd3969/attachment.html>

From mrosso at uci.edu  Thu Sep  3 12:16:34 2015
From: mrosso at uci.edu (Michele Rosso)
Date: Thu, 03 Sep 2015 10:16:34 -0700
Subject: [petsc-users] Problem installing metis
Message-ID: <1441300594.25982.2.camel@kolmog5>

Hi,

I am trying to install METIS via the development version of petsc, but I
encountered the following error:

         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
for details):
-------------------------------------------------------------------------------
Downloaded metis could not be used. Please check install
in /opt/petsc/petscfork/gnu-opt-32idx


I attached the install script and the configure.log file.
Any idea?

Thanks,
Michele



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/822f1a44/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 2184481 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/822f1a44/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gnu-opt-32idx.py
Type: text/x-python
Size: 927 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/822f1a44/attachment-0001.py>

From balay at mcs.anl.gov  Thu Sep  3 13:08:18 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 3 Sep 2015 13:08:18 -0500
Subject: [petsc-users] Problem installing metis
In-Reply-To: <1441300594.25982.2.camel@kolmog5>
References: <1441300594.25982.2.camel@kolmog5>
Message-ID: <alpine.LFD.2.20.1509031302410.6861@asterix>

Hm - this is weird.

>>>>>>>>>>>

TESTING: checkMath from config.libraries(config/BuildSystem/config/libraries.py:251)
  Check for sin() in libm, the math library

Executing: mpicc  -o /tmp/petsc-AXFj8c/config.libraries/conftest    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -ffast-math -march=native -mtune=native /tmp/petsc-AXFj8c/config.libraries/conftest.o   -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread -lgcc_s -ldl
              Popping language C
          Math functions are linked in by default


Executing: mpicc  -o /tmp/petsc-AXFj8c/config.libraries/conftest    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -ffast-math -march=native -mtune=native /tmp/petsc-AXFj8c/config.libraries/conftest.o  -Wl,-rpath,/opt/petsc/petscfork/gnu-opt-32idx/lib -L/opt/petsc/petscfork/gnu-opt-32idx/lib -lmetis -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread -lgcc_s -ldl
Possible ERROR while running linker: exit code 256
stderr:
/opt/petsc/petscfork/gnu-opt-32idx/lib/libmetis.a(kmetis.c.o): In function `libmetis__InitKWayPartitioning':
/opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/libmetis/kmetis.c:187: undefined reference to `__log_finite'
/opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/libmetis/kmetis.c:187: undefined reference to `__pow_finite'
/opt/petsc/petscfork/gnu-opt-32idx/lib/libmetis.a(util.c.o): In function `gk_flog2':
/opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/GKlib/util.c:106: undefined reference to `__log_finite'
<<<<<<<<<<

Can you rerun configure with the additional option "LIBS=-lm" - and see if it makes a difference?

Satish


On Thu, 3 Sep 2015, Michele Rosso wrote:

> Hi,
> 
> I am trying to install METIS via the development version of petsc, but I
> encountered the following error:
> 
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
> for details):
> -------------------------------------------------------------------------------
> Downloaded metis could not be used. Please check install
> in /opt/petsc/petscfork/gnu-opt-32idx
> 
> 
> I attached the install script and the configure.log file.
> Any idea?
> 
> Thanks,
> Michele
> 
> 
> 
> 


From mrosso at uci.edu  Thu Sep  3 13:16:27 2015
From: mrosso at uci.edu (Michele Rosso)
Date: Thu, 03 Sep 2015 11:16:27 -0700
Subject: [petsc-users] Problem installing metis
In-Reply-To: <alpine.LFD.2.20.1509031302410.6861@asterix>
References: <1441300594.25982.2.camel@kolmog5>
	<alpine.LFD.2.20.1509031302410.6861@asterix>
Message-ID: <1441304187.25982.4.camel@kolmog5>

Hi Satish,

I did as you suggested and I was able to install successfully.
I attached the configure script and log.

Thanks!
Michele

On Thu, 2015-09-03 at 13:08 -0500, Satish Balay wrote:

> Hm - this is weird.
> 
> >>>>>>>>>>>
> 
> TESTING: checkMath from config.libraries(config/BuildSystem/config/libraries.py:251)
>   Check for sin() in libm, the math library
> 
> Executing: mpicc  -o /tmp/petsc-AXFj8c/config.libraries/conftest    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -ffast-math -march=native -mtune=native /tmp/petsc-AXFj8c/config.libraries/conftest.o   -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread -lgcc_s -ldl
>               Popping language C
>           Math functions are linked in by default
> 
> 
> Executing: mpicc  -o /tmp/petsc-AXFj8c/config.libraries/conftest    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -ffast-math -march=native -mtune=native /tmp/petsc-AXFj8c/config.libraries/conftest.o  -Wl,-rpath,/opt/petsc/petscfork/gnu-opt-32idx/lib -L/opt/petsc/petscfork/gnu-opt-32idx/lib -lmetis -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread -lgcc_s -ldl
> Possible ERROR while running linker: exit code 256
> stderr:
> /opt/petsc/petscfork/gnu-opt-32idx/lib/libmetis.a(kmetis.c.o): In function `libmetis__InitKWayPartitioning':
> /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/libmetis/kmetis.c:187: undefined reference to `__log_finite'
> /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/libmetis/kmetis.c:187: undefined reference to `__pow_finite'
> /opt/petsc/petscfork/gnu-opt-32idx/lib/libmetis.a(util.c.o): In function `gk_flog2':
> /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/GKlib/util.c:106: undefined reference to `__log_finite'
> <<<<<<<<<<
> 
> Can you rerun configure with the additional option "LIBS=-lm" - and see if it makes a difference?
> 
> Satish
> 
> 
> On Thu, 3 Sep 2015, Michele Rosso wrote:
> 
> > Hi,
> > 
> > I am trying to install METIS via the development version of petsc, but I
> > encountered the following error:
> > 
> >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
> > for details):
> > -------------------------------------------------------------------------------
> > Downloaded metis could not be used. Please check install
> > in /opt/petsc/petscfork/gnu-opt-32idx
> > 
> > 
> > I attached the install script and the configure.log file.
> > Any idea?
> > 
> > Thanks,
> > Michele
> > 
> > 
> > 
> > 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/bf31d4c0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 3065355 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/bf31d4c0/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gnu-opt-32idx.py
Type: text/x-python
Size: 945 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/bf31d4c0/attachment-0001.py>

From bsmith at mcs.anl.gov  Thu Sep  3 13:24:42 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 Sep 2015 13:24:42 -0500
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
Message-ID: <5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>


    Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working in 3d)   You need to copy this routine and add whatever additional preallocation information you need.   Then call DMDASetGetMatrix()  so that the DM will use your routine to create the matrix for you.

   Barry




> On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com> wrote:
> 
> Hi,
> 
> I am using DMCreateMatrix to create matrix from a existed DM object and defined stencil.
> In my code, boundary nodes need to involve many inner nodes, thus matrix rows corresponding to boundary nodes are almost dense. How can I  tell petsc that those rows need to be preallocated with more entries? I don't want to use MatMPIAIJSetPreallocation() since advantages of DM might be lost.
> 
> Many thanks.
> 
> Sam.
> 
>  


From sghosh2012 at gatech.edu  Thu Sep  3 17:55:34 2015
From: sghosh2012 at gatech.edu (Ghosh, Swarnava)
Date: Thu, 3 Sep 2015 18:55:34 -0400 (EDT)
Subject: [petsc-users] Preserving nonzero structure during MatMatMult
In-Reply-To: <1282215395.2853606.1441318848644.JavaMail.root@mail.gatech.edu>
Message-ID: <130734507.2858113.1441320934695.JavaMail.root@mail.gatech.edu>

Hi,
 
  I want to do the following:
Matrices DM1, DM2 and DM3 (MPIAIJ type) have the same non zero structure. I want to multiply DM1*DM2=DM3. BUT the resulting matrix should preserve the initial non-zero pattern that was set to it. Basically multiplication should not create new non-zero locations.

For this I do:

/* after setting the nonzero locations/*

    ierr =  MatAssemblyBegin(DM1,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
    ierr =  MatAssemblyEnd(DM1,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
   
   MatSetOption(DM1,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE);
   MatSetOption(DM1,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
   
   
 Similarly for the matrices DM2 and DM3, the above is done.

However When I do MatMatMultSymbolic, the number of non-zeros change,

BEFORE MULT: NNZ allocated : 531441.000000 NNZ used: 518155.000000 
AFTER MULT: NNZ allocated : 2350209.000000 NNZ used: 2350209.000000 


Could you let me know how do I preserve the nonzero locations after MatMatMult?

Regards,
Swarnava
-- 
Swarnava Ghosh
PhD Candidate,
Structural Engineering, Mechanics and Materials
School of Civil and Environmental Engineering
Georgia Institute of Technology
Atlanta, GA 30332

From david.knezevic at akselos.com  Thu Sep  3 18:52:08 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Thu, 3 Sep 2015 19:52:08 -0400
Subject: [petsc-users] Adding rows and columns to a matrix
Message-ID: <CAJCWK9BHPAy7mXTiCKc5mtbKMBL_KjGUJa_APZhQGxEnDDTzjw@mail.gmail.com>

I need to add Lagrange multipliers to an existing system of equations,
which requires me to expand the matrix.

I figured I would allocate a new larger matrix with the appropriate
non-zero structure, and then copy the entries from the smaller matrix over
to it. But I was wondering if there might be a better way to do this?

Thanks,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/52602353/attachment.html>

From knepley at gmail.com  Thu Sep  3 18:57:28 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Sep 2015 19:57:28 -0400
Subject: [petsc-users] Preserving nonzero structure during MatMatMult
In-Reply-To: <130734507.2858113.1441320934695.JavaMail.root@mail.gatech.edu>
References: <1282215395.2853606.1441318848644.JavaMail.root@mail.gatech.edu>
	<130734507.2858113.1441320934695.JavaMail.root@mail.gatech.edu>
Message-ID: <CAMYG4GnZ5RsAAVUW9uh0dpX5yEJJqWmFWHpBjTmZq+xxASeRrA@mail.gmail.com>

On Thu, Sep 3, 2015 at 6:55 PM, Ghosh, Swarnava <sghosh2012 at gatech.edu>
wrote:

> Hi,
>
>   I want to do the following:
> Matrices DM1, DM2 and DM3 (MPIAIJ type) have the same non zero structure.
> I want to multiply DM1*DM2=DM3. BUT the resulting matrix should preserve
> the initial non-zero pattern that was set to it. Basically multiplication
> should not create new non-zero locations.
>
> For this I do:
>
> /* after setting the nonzero locations/*
>
>     ierr =  MatAssemblyBegin(DM1,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
>     ierr =  MatAssemblyEnd(DM1,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
>
>    MatSetOption(DM1,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE);
>    MatSetOption(DM1,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>
>
>  Similarly for the matrices DM2 and DM3, the above is done.
>
> However When I do MatMatMultSymbolic, the number of non-zeros change,
>
> BEFORE MULT: NNZ allocated : 531441.000000 NNZ used: 518155.000000
> AFTER MULT: NNZ allocated : 2350209.000000 NNZ used: 2350209.000000
>
>
> Could you let me know how do I preserve the nonzero locations after
> MatMatMult?
>

We do not support that. What is the motivation?

  Thanks,

     Matt


> Regards,
> Swarnava
> --
> Swarnava Ghosh
> PhD Candidate,
> Structural Engineering, Mechanics and Materials
> School of Civil and Environmental Engineering
> Georgia Institute of Technology
> Atlanta, GA 30332
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/765de781/attachment.html>

From knepley at gmail.com  Thu Sep  3 18:58:05 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Sep 2015 19:58:05 -0400
Subject: [petsc-users] Adding rows and columns to a matrix
In-Reply-To: <CAJCWK9BHPAy7mXTiCKc5mtbKMBL_KjGUJa_APZhQGxEnDDTzjw@mail.gmail.com>
References: <CAJCWK9BHPAy7mXTiCKc5mtbKMBL_KjGUJa_APZhQGxEnDDTzjw@mail.gmail.com>
Message-ID: <CAMYG4GkJ+K=MGX7E4rcwRu3Wi9AYdkDBP_CUFKr=dSqoWom8Eg@mail.gmail.com>

On Thu, Sep 3, 2015 at 7:52 PM, David Knezevic <david.knezevic at akselos.com>
wrote:

> I need to add Lagrange multipliers to an existing system of equations,
> which requires me to expand the matrix.
>
> I figured I would allocate a new larger matrix with the appropriate
> non-zero structure, and then copy the entries from the smaller matrix over
> to it. But I was wondering if there might be a better way to do this?
>

If you don't know the pattern at the beginning, its the best way.

   Thanks,

     Matt


> Thanks,
> David
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/73656be4/attachment.html>

From bsmith at mcs.anl.gov  Thu Sep  3 19:11:14 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 Sep 2015 19:11:14 -0500
Subject: [petsc-users] Adding rows and columns to a matrix
In-Reply-To: <CAJCWK9BHPAy7mXTiCKc5mtbKMBL_KjGUJa_APZhQGxEnDDTzjw@mail.gmail.com>
References: <CAJCWK9BHPAy7mXTiCKc5mtbKMBL_KjGUJa_APZhQGxEnDDTzjw@mail.gmail.com>
Message-ID: <5888D1FC-748A-4950-B0BC-51EE0EA79BB5@mcs.anl.gov>


  You might also consider the MatNest object. It's job is to help construct matrices that have a natural set of sub matrices such as yours. Also look at MatGetLocalSubMat()

  Barry


> On Sep 3, 2015, at 6:52 PM, David Knezevic <david.knezevic at akselos.com> wrote:
> 
> I need to add Lagrange multipliers to an existing system of equations, which requires me to expand the matrix.
> 
> I figured I would allocate a new larger matrix with the appropriate non-zero structure, and then copy the entries from the smaller matrix over to it. But I was wondering if there might be a better way to do this?
> 
> Thanks,
> David
> 
> 


From balay at mcs.anl.gov  Thu Sep  3 19:33:42 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 3 Sep 2015 19:33:42 -0500
Subject: [petsc-users] Problem installing metis
In-Reply-To: <1441304187.25982.4.camel@kolmog5>
References: <1441300594.25982.2.camel@kolmog5>
	<alpine.LFD.2.20.1509031302410.6861@asterix>
	<1441304187.25982.4.camel@kolmog5>
Message-ID: <alpine.LFD.2.20.1509031715100.8825@asterix>

Ok - the issue here is:- '-O3 -ffast-math' inlines/optimzies out the
math code in configure test [so -lm isn't needed/required here].  But
the code in metis [and petsc] library sources isn't inlined out
[requiring -lm].

>>>>>>>>>>>> Current configure behavior <<<<<<<<<<<
$ cat conftest.c 
double pow(double, double);
static void _check_pow() { double x = 0,y ; y = pow(x, x);
; }
int main() {
_check_pow();;
  return 0;
}
$ gcc -O -ffast-math conftest.c
$ gcc  conftest.c
/tmp/ccWCYvMy.o: In function `_check_pow':
conftest.c:(.text+0x28): undefined reference to `pow'
collect2: error: ld returned 1 exit status
$ gcc  conftest.c -lm
$ 
>>>>>>>>>>>> modified testcode to reproduce library-code behavior <<<<<<<<<<<
$ cat newtest.c 
#include <math.h>
//double pow(double, double); //also works but gives different error
double _check_pow(double x ) { return pow(x, x);
; }
int main() {
_check_pow(1.0);;
  return 0;
}
$ gcc -O3 -ffast-math newtest.c
/tmp/ccfeotgx.o: In function `_check_pow':
newtest.c:(.text+0x5): undefined reference to `__pow_finite'
collect2: error: ld returned 1 exit status
$ gcc -O3 -ffast-math newtest.c -lm
$ 
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

This test code within configure is spun by code-generator- so I'm not
sure how to modify it to get this test working..

[and its not clear if this change works universally - a different
compiler/optimizer might not fail the same way?]

Satish

On Thu, 3 Sep 2015, Michele Rosso wrote:

> Hi Satish,
> 
> I did as you suggested and I was able to install successfully.
> I attached the configure script and log.
> 
> Thanks!
> Michele
> 
> On Thu, 2015-09-03 at 13:08 -0500, Satish Balay wrote:
> 
> > Hm - this is weird.
> > 
> > >>>>>>>>>>>
> > 
> > TESTING: checkMath from config.libraries(config/BuildSystem/config/libraries.py:251)
> >   Check for sin() in libm, the math library
> > 
> > Executing: mpicc  -o /tmp/petsc-AXFj8c/config.libraries/conftest    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -ffast-math -march=native -mtune=native /tmp/petsc-AXFj8c/config.libraries/conftest.o   -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread -lgcc_s -ldl
> >               Popping language C
> >           Math functions are linked in by default
> > 
> > 
> > Executing: mpicc  -o /tmp/petsc-AXFj8c/config.libraries/conftest    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -ffast-math -march=native -mtune=native /tmp/petsc-AXFj8c/config.libraries/conftest.o  -Wl,-rpath,/opt/petsc/petscfork/gnu-opt-32idx/lib -L/opt/petsc/petscfork/gnu-opt-32idx/lib -lmetis -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread -lgcc_s -ldl
> > Possible ERROR while running linker: exit code 256
> > stderr:
> > /opt/petsc/petscfork/gnu-opt-32idx/lib/libmetis.a(kmetis.c.o): In function `libmetis__InitKWayPartitioning':
> > /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/libmetis/kmetis.c:187: undefined reference to `__log_finite'
> > /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/libmetis/kmetis.c:187: undefined reference to `__pow_finite'
> > /opt/petsc/petscfork/gnu-opt-32idx/lib/libmetis.a(util.c.o): In function `gk_flog2':
> > /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/GKlib/util.c:106: undefined reference to `__log_finite'
> > <<<<<<<<<<
> > 
> > Can you rerun configure with the additional option "LIBS=-lm" - and see if it makes a difference?
> > 
> > Satish
> > 
> > 
> > On Thu, 3 Sep 2015, Michele Rosso wrote:
> > 
> > > Hi,
> > > 
> > > I am trying to install METIS via the development version of petsc, but I
> > > encountered the following error:
> > > 
> > >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
> > > for details):
> > > -------------------------------------------------------------------------------
> > > Downloaded metis could not be used. Please check install
> > > in /opt/petsc/petscfork/gnu-opt-32idx
> > > 
> > > 
> > > I attached the install script and the configure.log file.
> > > Any idea?
> > > 
> > > Thanks,
> > > Michele
> > > 
> > > 
> > > 
> > > 
> > 
> 
> 
> 


From bsmith at mcs.anl.gov  Thu Sep  3 19:55:15 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 Sep 2015 19:55:15 -0500
Subject: [petsc-users] Problem installing metis
In-Reply-To: <alpine.LFD.2.20.1509031715100.8825@asterix>
References: <1441300594.25982.2.camel@kolmog5>
	<alpine.LFD.2.20.1509031302410.6861@asterix>
	<1441304187.25982.4.camel@kolmog5>
	<alpine.LFD.2.20.1509031715100.8825@asterix>
Message-ID: <FEA36935-DF4D-47CE-8142-ED8954DF192F@mcs.anl.gov>


  Satish,

   Well the classic model that Matt inherited from GNU that does not include the appropriate include files but instead makes up some prototype in the test file and looks for symbols in the resulting .o is not IMHO good anyways and I'd like to see it eventually totally eliminated so here is one place it can be done.  Use the checklink() function that takes an include and a main body to check these math functions?  If you have the main() return depend on the result of the math function then it cannot simply remove the entire call.


  Barry


> On Sep 3, 2015, at 7:33 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> Ok - the issue here is:- '-O3 -ffast-math' inlines/optimzies out the
> math code in configure test [so -lm isn't needed/required here].  But
> the code in metis [and petsc] library sources isn't inlined out
> [requiring -lm].
> 
>>>>>>>>>>>>> Current configure behavior <<<<<<<<<<<
> $ cat conftest.c 
> double pow(double, double);
> static void _check_pow() { double x = 0,y ; y = pow(x, x);
> ; }
> int main() {
> _check_pow();;
>  return 0;
> }
> $ gcc -O -ffast-math conftest.c
> $ gcc  conftest.c
> /tmp/ccWCYvMy.o: In function `_check_pow':
> conftest.c:(.text+0x28): undefined reference to `pow'
> collect2: error: ld returned 1 exit status
> $ gcc  conftest.c -lm
> $ 
>>>>>>>>>>>>> modified testcode to reproduce library-code behavior <<<<<<<<<<<
> $ cat newtest.c 
> #include <math.h>
> //double pow(double, double); //also works but gives different error
> double _check_pow(double x ) { return pow(x, x);
> ; }
> int main() {
> _check_pow(1.0);;
>  return 0;
> }
> $ gcc -O3 -ffast-math newtest.c
> /tmp/ccfeotgx.o: In function `_check_pow':
> newtest.c:(.text+0x5): undefined reference to `__pow_finite'
> collect2: error: ld returned 1 exit status
> $ gcc -O3 -ffast-math newtest.c -lm
> $ 
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> 
> This test code within configure is spun by code-generator- so I'm not
> sure how to modify it to get this test working..
> 
> [and its not clear if this change works universally - a different
> compiler/optimizer might not fail the same way?]
> 
> Satish
> 
> On Thu, 3 Sep 2015, Michele Rosso wrote:
> 
>> Hi Satish,
>> 
>> I did as you suggested and I was able to install successfully.
>> I attached the configure script and log.
>> 
>> Thanks!
>> Michele
>> 
>> On Thu, 2015-09-03 at 13:08 -0500, Satish Balay wrote:
>> 
>>> Hm - this is weird.
>>> 
>>>>>>>>>>>>>> 
>>> 
>>> TESTING: checkMath from config.libraries(config/BuildSystem/config/libraries.py:251)
>>>  Check for sin() in libm, the math library
>>> 
>>> Executing: mpicc  -o /tmp/petsc-AXFj8c/config.libraries/conftest    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -ffast-math -march=native -mtune=native /tmp/petsc-AXFj8c/config.libraries/conftest.o   -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread -lgcc_s -ldl
>>>              Popping language C
>>>          Math functions are linked in by default
>>> 
>>> 
>>> Executing: mpicc  -o /tmp/petsc-AXFj8c/config.libraries/conftest    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -ffast-math -march=native -mtune=native /tmp/petsc-AXFj8c/config.libraries/conftest.o  -Wl,-rpath,/opt/petsc/petscfork/gnu-opt-32idx/lib -L/opt/petsc/petscfork/gnu-opt-32idx/lib -lmetis -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread -lgcc_s -ldl
>>> Possible ERROR while running linker: exit code 256
>>> stderr:
>>> /opt/petsc/petscfork/gnu-opt-32idx/lib/libmetis.a(kmetis.c.o): In function `libmetis__InitKWayPartitioning':
>>> /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/libmetis/kmetis.c:187: undefined reference to `__log_finite'
>>> /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/libmetis/kmetis.c:187: undefined reference to `__pow_finite'
>>> /opt/petsc/petscfork/gnu-opt-32idx/lib/libmetis.a(util.c.o): In function `gk_flog2':
>>> /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/GKlib/util.c:106: undefined reference to `__log_finite'
>>> <<<<<<<<<<
>>> 
>>> Can you rerun configure with the additional option "LIBS=-lm" - and see if it makes a difference?
>>> 
>>> Satish
>>> 
>>> 
>>> On Thu, 3 Sep 2015, Michele Rosso wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I am trying to install METIS via the development version of petsc, but I
>>>> encountered the following error:
>>>> 
>>>>         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
>>>> for details):
>>>> -------------------------------------------------------------------------------
>>>> Downloaded metis could not be used. Please check install
>>>> in /opt/petsc/petscfork/gnu-opt-32idx
>>>> 
>>>> 
>>>> I attached the install script and the configure.log file.
>>>> Any idea?
>>>> 
>>>> Thanks,
>>>> Michele
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
> 


From david.knezevic at akselos.com  Thu Sep  3 20:17:28 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Thu, 3 Sep 2015 21:17:28 -0400
Subject: [petsc-users] Adding rows and columns to a matrix
In-Reply-To: <5888D1FC-748A-4950-B0BC-51EE0EA79BB5@mcs.anl.gov>
References: <CAJCWK9BHPAy7mXTiCKc5mtbKMBL_KjGUJa_APZhQGxEnDDTzjw@mail.gmail.com>
	<5888D1FC-748A-4950-B0BC-51EE0EA79BB5@mcs.anl.gov>
Message-ID: <CAJCWK9DYMvm4VnFwU71dg0WbcP9YN6A8UqEjBi3rjwnLP1K--g@mail.gmail.com>

On Thu, Sep 3, 2015 at 8:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   You might also consider the MatNest object. It's job is to help
> construct matrices that have a natural set of sub matrices such as yours.
> Also look at MatGetLocalSubMat()
>


OK, thanks for the pointers.

Best regards,
David




>
> > On Sep 3, 2015, at 6:52 PM, David Knezevic <david.knezevic at akselos.com>
> wrote:
> >
> > I need to add Lagrange multipliers to an existing system of equations,
> which requires me to expand the matrix.
> >
> > I figured I would allocate a new larger matrix with the appropriate
> non-zero structure, and then copy the entries from the smaller matrix over
> to it. But I was wondering if there might be a better way to do this?
> >
> > Thanks,
> > David
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/eabf033d/attachment.html>

From balay at mcs.anl.gov  Thu Sep  3 20:21:46 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 3 Sep 2015 20:21:46 -0500
Subject: [petsc-users] Problem installing metis
In-Reply-To: <FEA36935-DF4D-47CE-8142-ED8954DF192F@mcs.anl.gov>
References: <1441300594.25982.2.camel@kolmog5>
	<alpine.LFD.2.20.1509031302410.6861@asterix>
	<1441304187.25982.4.camel@kolmog5>
	<alpine.LFD.2.20.1509031715100.8825@asterix>
	<FEA36935-DF4D-47CE-8142-ED8954DF192F@mcs.anl.gov>
Message-ID: <alpine.LFD.2.20.1509032011590.8772@asterix>

The primary issue is not the include statement - that just gives a
different error message:

undefined reference to `__pow_finite'
vs
undefined reference to `pow'

The issue is:
static void _check_pow() { double x = 0,y ; y = pow(x, x);
vs
double _check_pow(double x ) { return pow(x, x);

i.e
1. remove static
2. pass in input 'double x' [instead of declaring in function]
3. pass back the retrun as 'dobule' - instead of locally computing value [with void]

config/BuildSystem/config/libraries.py uses code generator to generate
a bunch of configure tests to generate the following test code for -lm
check [and uses the same genater for abunch of other tests]

Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
double sin(double);
static void _check_sin() { double x = 0,y; y = sin(x);
; }
double floor(double);
static void _check_floor() { double x = 0,y; y = floor(x);
; }
double log10(double);
static void _check_log10() { double x = 0,y; y = log10(x);
; }
double pow(double, double);
static void _check_pow() { double x = 0,y ; y = pow(x, x);
; }

int main() {
_check_sin();
_check_floor();
_check_log10();
_check_pow();;
  return 0;
}

And there are comments like:
>>>>>>>
      # Capture the function call in a static function so that any local variables are isolated from
      # calls to other library functions.
<<<<<<<<

So we'll have to look at this code generater [and its assumtions] -
and see if it can be modified to generate this alternate code..

Satish

On Thu, 3 Sep 2015, Barry Smith wrote:

> 
>   Satish,
> 
>    Well the classic model that Matt inherited from GNU that does not include the appropriate include files but instead makes up some prototype in the test file and looks for symbols in the resulting .o is not IMHO good anyways and I'd like to see it eventually totally eliminated so here is one place it can be done.  Use the checklink() function that takes an include and a main body to check these math functions?  If you have the main() return depend on the result of the math function then it cannot simply remove the entire call.
> 
> 
>   Barry
> 
> 
> > On Sep 3, 2015, at 7:33 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> > 
> > Ok - the issue here is:- '-O3 -ffast-math' inlines/optimzies out the
> > math code in configure test [so -lm isn't needed/required here].  But
> > the code in metis [and petsc] library sources isn't inlined out
> > [requiring -lm].
> > 
> >>>>>>>>>>>>> Current configure behavior <<<<<<<<<<<
> > $ cat conftest.c 
> > double pow(double, double);
> > static void _check_pow() { double x = 0,y ; y = pow(x, x);
> > ; }
> > int main() {
> > _check_pow();;
> >  return 0;
> > }
> > $ gcc -O -ffast-math conftest.c
> > $ gcc  conftest.c
> > /tmp/ccWCYvMy.o: In function `_check_pow':
> > conftest.c:(.text+0x28): undefined reference to `pow'
> > collect2: error: ld returned 1 exit status
> > $ gcc  conftest.c -lm
> > $ 
> >>>>>>>>>>>>> modified testcode to reproduce library-code behavior <<<<<<<<<<<
> > $ cat newtest.c 
> > #include <math.h>
> > //double pow(double, double); //also works but gives different error
> > double _check_pow(double x ) { return pow(x, x);
> > ; }
> > int main() {
> > _check_pow(1.0);;
> >  return 0;
> > }
> > $ gcc -O3 -ffast-math newtest.c
> > /tmp/ccfeotgx.o: In function `_check_pow':
> > newtest.c:(.text+0x5): undefined reference to `__pow_finite'
> > collect2: error: ld returned 1 exit status
> > $ gcc -O3 -ffast-math newtest.c -lm
> > $ 
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > 
> > This test code within configure is spun by code-generator- so I'm not
> > sure how to modify it to get this test working..
> > 
> > [and its not clear if this change works universally - a different
> > compiler/optimizer might not fail the same way?]
> > 
> > Satish
> > 
> > On Thu, 3 Sep 2015, Michele Rosso wrote:
> > 
> >> Hi Satish,
> >> 
> >> I did as you suggested and I was able to install successfully.
> >> I attached the configure script and log.
> >> 
> >> Thanks!
> >> Michele
> >> 
> >> On Thu, 2015-09-03 at 13:08 -0500, Satish Balay wrote:
> >> 
> >>> Hm - this is weird.
> >>> 
> >>>>>>>>>>>>>> 
> >>> 
> >>> TESTING: checkMath from config.libraries(config/BuildSystem/config/libraries.py:251)
> >>>  Check for sin() in libm, the math library
> >>> 
> >>> Executing: mpicc  -o /tmp/petsc-AXFj8c/config.libraries/conftest    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -ffast-math -march=native -mtune=native /tmp/petsc-AXFj8c/config.libraries/conftest.o   -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread -lgcc_s -ldl
> >>>              Popping language C
> >>>          Math functions are linked in by default
> >>> 
> >>> 
> >>> Executing: mpicc  -o /tmp/petsc-AXFj8c/config.libraries/conftest    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -ffast-math -march=native -mtune=native /tmp/petsc-AXFj8c/config.libraries/conftest.o  -Wl,-rpath,/opt/petsc/petscfork/gnu-opt-32idx/lib -L/opt/petsc/petscfork/gnu-opt-32idx/lib -lmetis -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread -lgcc_s -ldl
> >>> Possible ERROR while running linker: exit code 256
> >>> stderr:
> >>> /opt/petsc/petscfork/gnu-opt-32idx/lib/libmetis.a(kmetis.c.o): In function `libmetis__InitKWayPartitioning':
> >>> /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/libmetis/kmetis.c:187: undefined reference to `__log_finite'
> >>> /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/libmetis/kmetis.c:187: undefined reference to `__pow_finite'
> >>> /opt/petsc/petscfork/gnu-opt-32idx/lib/libmetis.a(util.c.o): In function `gk_flog2':
> >>> /opt/petsc/petscfork/gnu-opt-32idx/externalpackages/metis-5.1.0-p1/GKlib/util.c:106: undefined reference to `__log_finite'
> >>> <<<<<<<<<<
> >>> 
> >>> Can you rerun configure with the additional option "LIBS=-lm" - and see if it makes a difference?
> >>> 
> >>> Satish
> >>> 
> >>> 
> >>> On Thu, 3 Sep 2015, Michele Rosso wrote:
> >>> 
> >>>> Hi,
> >>>> 
> >>>> I am trying to install METIS via the development version of petsc, but I
> >>>> encountered the following error:
> >>>> 
> >>>>         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
> >>>> for details):
> >>>> -------------------------------------------------------------------------------
> >>>> Downloaded metis could not be used. Please check install
> >>>> in /opt/petsc/petscfork/gnu-opt-32idx
> >>>> 
> >>>> 
> >>>> I attached the install script and the configure.log file.
> >>>> Any idea?
> >>>> 
> >>>> Thanks,
> >>>> Michele
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>> 
> >> 
> >> 
> >> 
> > 
> 
> 


From sghosh2012 at gatech.edu  Thu Sep  3 20:31:06 2015
From: sghosh2012 at gatech.edu (Ghosh, Swarnava)
Date: Thu, 3 Sep 2015 21:31:06 -0400 (EDT)
Subject: [petsc-users] Preserving nonzero structure during MatMatMult
In-Reply-To: <CAMYG4GnZ5RsAAVUW9uh0dpX5yEJJqWmFWHpBjTmZq+xxASeRrA@mail.gmail.com>
Message-ID: <1955987296.2899288.1441330266178.JavaMail.root@mail.gatech.edu>

Hi Matt, 


The motivation is to calculate density matrix in electronic structure calculations. 
The density matrix can be obtained recursively iterations (known as density matrix purification). An example is 


X_{n+1} = 3 X_n^2 - 2 X_n^3 


Density matrix has property of off diagonal decay. 


To do this I was planning to setup the non-zero structure of these matrices using the decay estimates and then proceed with the iteration such that the non-zero structure of the product matrices are preserved. In doing so I would need to do MatMatMultSymbolic once. 


Another option I thought of is to truncate the values in the product matrix after MatMatMultNumeric. However this would require to do MatMatMultSymbolic for every subsequent matrix product and also matrix assembly after every truncation and would not be very efficient. 


Please let me know if you have any other thoughts regarding this. 




Regards. 
Swarnava 
----- Original Message -----

From: "Matthew Knepley" <knepley at gmail.com> 
To: "Swarnava Ghosh" <sghosh2012 at gatech.edu> 
Cc: "PETSc users list" <petsc-users at mcs.anl.gov> 
Sent: Thursday, September 3, 2015 7:57:28 PM 
Subject: Re: [petsc-users] Preserving nonzero structure during MatMatMult 




On Thu, Sep 3, 2015 at 6:55 PM, Ghosh, Swarnava < sghosh2012 at gatech.edu > wrote: 


Hi, 

I want to do the following: 
Matrices DM1, DM2 and DM3 (MPIAIJ type) have the same non zero structure. I want to multiply DM1*DM2=DM3. BUT the resulting matrix should preserve the initial non-zero pattern that was set to it. Basically multiplication should not create new non-zero locations. 

For this I do: 

/* after setting the nonzero locations/* 

ierr = MatAssemblyBegin(DM1,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); 
ierr = MatAssemblyEnd(DM1,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); 

MatSetOption(DM1,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); 
MatSetOption(DM1,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); 


Similarly for the matrices DM2 and DM3, the above is done. 

However When I do MatMatMultSymbolic, the number of non-zeros change, 

BEFORE MULT: NNZ allocated : 531441.000000 NNZ used: 518155.000000 
AFTER MULT: NNZ allocated : 2350209.000000 NNZ used: 2350209.000000 


Could you let me know how do I preserve the nonzero locations after MatMatMult? 





We do not support that. What is the motivation? 


Thanks, 


Matt 

<blockquote>
Regards, 
Swarnava 
-- 
Swarnava Ghosh 
PhD Candidate, 
Structural Engineering, Mechanics and Materials 
School of Civil and Environmental Engineering 
Georgia Institute of Technology 
Atlanta, GA 30332 

</blockquote>




-- 

What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 


-- 

Swarnava Ghosh 
PhD Candidate, 
Structural Engineering, Mechanics and Materials 
School of Civil and Environmental Engineering 
Georgia Institute of Technology 
Atlanta, GA 30332 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150903/beffad42/attachment.html>

From timothee.nicolas at gmail.com  Fri Sep  4 10:20:07 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Sat, 5 Sep 2015 00:20:07 +0900
Subject: [petsc-users] Most efficient way to set a matrix representing the
	linearized operator
Message-ID: <CAGi1ndTG7ifvNUn7vCBqryyhD-vwdCWGg1BSB6VSR=DqEk0_RQ@mail.gmail.com>

Hi,

(short version at the end)

I am starting to write the matrices for the preconditioner of my
application. It's a physics-based preconditioner (L Chacon Journal of
Comput. Phys.  2002, 2003 and Physics of Plasmas 2010 if you want to know
more), so I must compute the linearized operators as matrices. I then plan
to use KSP Multigrid to invert the resulting linear systems.

My question is, what is the most efficient way of setting the matrices for
linearized operators in the context of a 3D grid handled with DMDA and
several degrees of freedom (8 in my case) ? Take for instance a term like
curl(v x B) in the induction equation, which, linearized, gives curl(v0 x .
) for the magnetic part. I think the logical approach is to use vectors of
the form (0,...,0,1,0,...,0) and to compute the result of the operator, and
store it in the matrix by columns using MatSetValuesStencil. The matrix is
created with DMCreateMatrix.

Since all the vectors have mostly zeros, by writing by hand what parts you
need to compute, the number of evaluations for each vector is only the
number of nonzeros in a column, typically 21 in my case (7 point stencil *
3 directions), so the total number of evaluations of the operator should be
reasonable and efficient that way.

It gives promising results, however I am having trouble handling the
regions at the interface between the processors. Because if your vector has
its only non zero value on a grid point which is on the boundary between
two processors, then it means that one of the non-zero value in the result
vector will be living on a ghost point. And to compute the operator at this
ghost point, you need an *extra* ghost point, which makes the whole thing
quite cumbersome. You could of course increase the stencil_width of the
DMDA by one, but it does not seem the good approach.

Instead, I was wondering if it would be possible to write the matrix by
column vectors. So I would still create vectors of the form
(0,...,0,1,0,...,0) but in Petsc Vec type, then compute the result directly
in Petsc Vec, and put this vector in the matrix as a column. But somehow it
does not seem really efficient, unless there is a way Petsc knows that the
vector you input has mostly zeros, as well as the result vector.

So, in short

1. How would you compute a matrix representing an operator in a 3D system
with several degrees of freedom in the most efficient way ?

And two other question related to this matter:

2. When I do the Multigrid inversion, I will momentarily ignore 3 degrees
of freedom (the velocity). What is the more efficient between (i) putting 1
everywhere on the diagonal in the velocity part or (ii) somehow create a
new DMDA with 8-3 = 5 degrees of freedom to handle the problem separately.

3. When I use DMCreateMatrix, do I have to preallocate the matrix ? I have
tried, but got errors like
[11]PETSC ERROR: New nonzero at (296,296) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
off this check

which could not be prevented by using the advised MatSetOption(A,
MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)

Sorry for the long question

Best

Timoth?e
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150905/3da78637/attachment.html>

From knepley at gmail.com  Fri Sep  4 13:42:16 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 Sep 2015 14:42:16 -0400
Subject: [petsc-users] Most efficient way to set a matrix representing
 the linearized operator
In-Reply-To: <CAGi1ndTG7ifvNUn7vCBqryyhD-vwdCWGg1BSB6VSR=DqEk0_RQ@mail.gmail.com>
References: <CAGi1ndTG7ifvNUn7vCBqryyhD-vwdCWGg1BSB6VSR=DqEk0_RQ@mail.gmail.com>
Message-ID: <CAMYG4GmAcr9__Y0g_RUY5w=WU0o4PuoFA=oaHL1ccVA7jgF4ww@mail.gmail.com>

On Fri, Sep 4, 2015 at 11:20 AM, Timoth?e Nicolas <
timothee.nicolas at gmail.com> wrote:

> Hi,
>
> (short version at the end)
>
> I am starting to write the matrices for the preconditioner of my
> application. It's a physics-based preconditioner (L Chacon Journal of
> Comput. Phys.  2002, 2003 and Physics of Plasmas 2010 if you want to know
> more), so I must compute the linearized operators as matrices. I then plan
> to use KSP Multigrid to invert the resulting linear systems.
>
> My question is, what is the most efficient way of setting the matrices for
> linearized operators in the context of a 3D grid handled with DMDA and
> several degrees of freedom (8 in my case) ? Take for instance a term like
> curl(v x B) in the induction equation, which, linearized, gives curl(v0 x .
> ) for the magnetic part. I think the logical approach is to use vectors of
> the form (0,...,0,1,0,...,0) and to compute the result of the operator, and
> store it in the matrix by columns using MatSetValuesStencil. The matrix is
> created with DMCreateMatrix.
>
> Since all the vectors have mostly zeros, by writing by hand what parts you
> need to compute, the number of evaluations for each vector is only the
> number of nonzeros in a column, typically 21 in my case (7 point stencil *
> 3 directions), so the total number of evaluations of the operator should be
> reasonable and efficient that way.
>
> It gives promising results, however I am having trouble handling the
> regions at the interface between the processors. Because if your vector has
> its only non zero value on a grid point which is on the boundary between
> two processors, then it means that one of the non-zero value in the result
> vector will be living on a ghost point. And to compute the operator at this
> ghost point, you need an *extra* ghost point, which makes the whole thing
> quite cumbersome. You could of course increase the stencil_width of the
> DMDA by one, but it does not seem the good approach.
>

Why not? The only reason that the stencil width exists is to specify how
much data from other processors
is necessary to compute the residual/Jacobian. It sounds like two ghosts
are needed.

  Thanks,

    Matt


> Instead, I was wondering if it would be possible to write the matrix by
> column vectors. So I would still create vectors of the form
> (0,...,0,1,0,...,0) but in Petsc Vec type, then compute the result directly
> in Petsc Vec, and put this vector in the matrix as a column. But somehow it
> does not seem really efficient, unless there is a way Petsc knows that the
> vector you input has mostly zeros, as well as the result vector.
>
> So, in short
>
> 1. How would you compute a matrix representing an operator in a 3D system
> with several degrees of freedom in the most efficient way ?
>
> And two other question related to this matter:
>
> 2. When I do the Multigrid inversion, I will momentarily ignore 3 degrees
> of freedom (the velocity). What is the more efficient between (i) putting 1
> everywhere on the diagonal in the velocity part or (ii) somehow create a
> new DMDA with 8-3 = 5 degrees of freedom to handle the problem separately.
>
> 3. When I use DMCreateMatrix, do I have to preallocate the matrix ? I have
> tried, but got errors like
> [11]PETSC ERROR: New nonzero at (296,296) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
> off this check
>
> which could not be prevented by using the advised MatSetOption(A,
> MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>
> Sorry for the long question
>
> Best
>
> Timoth?e
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150904/e8d74ef4/attachment.html>

From timothee.nicolas at gmail.com  Fri Sep  4 19:20:53 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Sat, 5 Sep 2015 09:20:53 +0900
Subject: [petsc-users] Most efficient way to set a matrix representing
 the linearized operator
In-Reply-To: <CAMYG4GmAcr9__Y0g_RUY5w=WU0o4PuoFA=oaHL1ccVA7jgF4ww@mail.gmail.com>
References: <CAGi1ndTG7ifvNUn7vCBqryyhD-vwdCWGg1BSB6VSR=DqEk0_RQ@mail.gmail.com>
	<CAMYG4GmAcr9__Y0g_RUY5w=WU0o4PuoFA=oaHL1ccVA7jgF4ww@mail.gmail.com>
Message-ID: <CAGi1ndQ9+UzCcQXZmCSX4ojXB3CyHMyWoxVX3q5bK7=bo+xNDQ@mail.gmail.com>

I see, good. Then there should be no problem. Do you also have a suggestion
for questions 2. and 3. ?

Timoth?e

2015-09-05 3:42 GMT+09:00 Matthew Knepley <knepley at gmail.com>:

> On Fri, Sep 4, 2015 at 11:20 AM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
>
>> Hi,
>>
>> (short version at the end)
>>
>> I am starting to write the matrices for the preconditioner of my
>> application. It's a physics-based preconditioner (L Chacon Journal of
>> Comput. Phys.  2002, 2003 and Physics of Plasmas 2010 if you want to know
>> more), so I must compute the linearized operators as matrices. I then plan
>> to use KSP Multigrid to invert the resulting linear systems.
>>
>> My question is, what is the most efficient way of setting the matrices
>> for linearized operators in the context of a 3D grid handled with DMDA and
>> several degrees of freedom (8 in my case) ? Take for instance a term like
>> curl(v x B) in the induction equation, which, linearized, gives curl(v0 x .
>> ) for the magnetic part. I think the logical approach is to use vectors of
>> the form (0,...,0,1,0,...,0) and to compute the result of the operator, and
>> store it in the matrix by columns using MatSetValuesStencil. The matrix is
>> created with DMCreateMatrix.
>>
>> Since all the vectors have mostly zeros, by writing by hand what parts
>> you need to compute, the number of evaluations for each vector is only the
>> number of nonzeros in a column, typically 21 in my case (7 point stencil *
>> 3 directions), so the total number of evaluations of the operator should be
>> reasonable and efficient that way.
>>
>> It gives promising results, however I am having trouble handling the
>> regions at the interface between the processors. Because if your vector has
>> its only non zero value on a grid point which is on the boundary between
>> two processors, then it means that one of the non-zero value in the result
>> vector will be living on a ghost point. And to compute the operator at this
>> ghost point, you need an *extra* ghost point, which makes the whole thing
>> quite cumbersome. You could of course increase the stencil_width of the
>> DMDA by one, but it does not seem the good approach.
>>
>
> Why not? The only reason that the stencil width exists is to specify how
> much data from other processors
> is necessary to compute the residual/Jacobian. It sounds like two ghosts
> are needed.
>
>   Thanks,
>
>     Matt
>
>
>> Instead, I was wondering if it would be possible to write the matrix by
>> column vectors. So I would still create vectors of the form
>> (0,...,0,1,0,...,0) but in Petsc Vec type, then compute the result directly
>> in Petsc Vec, and put this vector in the matrix as a column. But somehow it
>> does not seem really efficient, unless there is a way Petsc knows that the
>> vector you input has mostly zeros, as well as the result vector.
>>
>> So, in short
>>
>> 1. How would you compute a matrix representing an operator in a 3D system
>> with several degrees of freedom in the most efficient way ?
>>
>> And two other question related to this matter:
>>
>> 2. When I do the Multigrid inversion, I will momentarily ignore 3 degrees
>> of freedom (the velocity). What is the more efficient between (i) putting 1
>> everywhere on the diagonal in the velocity part or (ii) somehow create a
>> new DMDA with 8-3 = 5 degrees of freedom to handle the problem separately.
>>
>> 3. When I use DMCreateMatrix, do I have to preallocate the matrix ? I
>> have tried, but got errors like
>> [11]PETSC ERROR: New nonzero at (296,296) caused a malloc
>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
>> off this check
>>
>> which could not be prevented by using the advised MatSetOption(A,
>> MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>>
>> Sorry for the long question
>>
>> Best
>>
>> Timoth?e
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150905/cf6ec987/attachment.html>

From knepley at gmail.com  Fri Sep  4 19:35:48 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 Sep 2015 20:35:48 -0400
Subject: [petsc-users] Most efficient way to set a matrix representing
 the linearized operator
In-Reply-To: <CAGi1ndQ9+UzCcQXZmCSX4ojXB3CyHMyWoxVX3q5bK7=bo+xNDQ@mail.gmail.com>
References: <CAGi1ndTG7ifvNUn7vCBqryyhD-vwdCWGg1BSB6VSR=DqEk0_RQ@mail.gmail.com>
	<CAMYG4GmAcr9__Y0g_RUY5w=WU0o4PuoFA=oaHL1ccVA7jgF4ww@mail.gmail.com>
	<CAGi1ndQ9+UzCcQXZmCSX4ojXB3CyHMyWoxVX3q5bK7=bo+xNDQ@mail.gmail.com>
Message-ID: <CAMYG4GmNPGBxo+9UKCqX7=7=vxfg8S_qEaR_7T1CW=H4LhtE-Q@mail.gmail.com>

On Fri, Sep 4, 2015 at 8:20 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com
> wrote:

> I see, good. Then there should be no problem. Do you also have a
> suggestion for questions 2. and 3. ?
>

2. Makes little difference. Get the solver converging and scaling first.

3. The DMDA allocates the matrix correctly for the stencil you specify. You
should not have the error
    if your insertions match the specification.

  Thanks,

    Matt


> Timoth?e
>
> 2015-09-05 3:42 GMT+09:00 Matthew Knepley <knepley at gmail.com>:
>
>> On Fri, Sep 4, 2015 at 11:20 AM, Timoth?e Nicolas <
>> timothee.nicolas at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> (short version at the end)
>>>
>>> I am starting to write the matrices for the preconditioner of my
>>> application. It's a physics-based preconditioner (L Chacon Journal of
>>> Comput. Phys.  2002, 2003 and Physics of Plasmas 2010 if you want to know
>>> more), so I must compute the linearized operators as matrices. I then plan
>>> to use KSP Multigrid to invert the resulting linear systems.
>>>
>>> My question is, what is the most efficient way of setting the matrices
>>> for linearized operators in the context of a 3D grid handled with DMDA and
>>> several degrees of freedom (8 in my case) ? Take for instance a term like
>>> curl(v x B) in the induction equation, which, linearized, gives curl(v0 x .
>>> ) for the magnetic part. I think the logical approach is to use vectors of
>>> the form (0,...,0,1,0,...,0) and to compute the result of the operator, and
>>> store it in the matrix by columns using MatSetValuesStencil. The matrix is
>>> created with DMCreateMatrix.
>>>
>>> Since all the vectors have mostly zeros, by writing by hand what parts
>>> you need to compute, the number of evaluations for each vector is only the
>>> number of nonzeros in a column, typically 21 in my case (7 point stencil *
>>> 3 directions), so the total number of evaluations of the operator should be
>>> reasonable and efficient that way.
>>>
>>> It gives promising results, however I am having trouble handling the
>>> regions at the interface between the processors. Because if your vector has
>>> its only non zero value on a grid point which is on the boundary between
>>> two processors, then it means that one of the non-zero value in the result
>>> vector will be living on a ghost point. And to compute the operator at this
>>> ghost point, you need an *extra* ghost point, which makes the whole thing
>>> quite cumbersome. You could of course increase the stencil_width of the
>>> DMDA by one, but it does not seem the good approach.
>>>
>>
>> Why not? The only reason that the stencil width exists is to specify how
>> much data from other processors
>> is necessary to compute the residual/Jacobian. It sounds like two ghosts
>> are needed.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Instead, I was wondering if it would be possible to write the matrix by
>>> column vectors. So I would still create vectors of the form
>>> (0,...,0,1,0,...,0) but in Petsc Vec type, then compute the result directly
>>> in Petsc Vec, and put this vector in the matrix as a column. But somehow it
>>> does not seem really efficient, unless there is a way Petsc knows that the
>>> vector you input has mostly zeros, as well as the result vector.
>>>
>>> So, in short
>>>
>>> 1. How would you compute a matrix representing an operator in a 3D
>>> system with several degrees of freedom in the most efficient way ?
>>>
>>> And two other question related to this matter:
>>>
>>> 2. When I do the Multigrid inversion, I will momentarily ignore 3
>>> degrees of freedom (the velocity). What is the more efficient between (i)
>>> putting 1 everywhere on the diagonal in the velocity part or (ii) somehow
>>> create a new DMDA with 8-3 = 5 degrees of freedom to handle the problem
>>> separately.
>>>
>>> 3. When I use DMCreateMatrix, do I have to preallocate the matrix ? I
>>> have tried, but got errors like
>>> [11]PETSC ERROR: New nonzero at (296,296) caused a malloc
>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
>>> off this check
>>>
>>> which could not be prevented by using the advised MatSetOption(A,
>>> MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>>>
>>> Sorry for the long question
>>>
>>> Best
>>>
>>> Timoth?e
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150904/dfac24d0/attachment-0001.html>

From gideon.simpson at gmail.com  Mon Sep  7 18:32:39 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 7 Sep 2015 19:32:39 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
Message-ID: <ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>

Barry,

I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?

-gideon

> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>> 
>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>> 
>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
> 
>   I would do the following. Create your DM and create a SNES that will do the continuation
> 
>   loop over continuation parameter
> 
>        SNESSolve(snes,NULL,Ucoarse);
> 
>        if (you decide you want to see the refined solution at this continuation point) {
>             SNESCreate(comm,&snesrefine);
>             SNESSetDM()
>             etc
>             SNESSetGridSequence(snesrefine,)
>             SNESSolve(snesrefine,0,Ucoarse);
>             SNESGetSolution(snesrefine,&Ufine);
>             VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>             SNESDestroy(snesrefine);
>       end if
> 
>   end loop over continuation parameter.
> 
>   Barry
> 
>> 
>> -gideon
>> 
>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>>> 
>>>> 
>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>> 
>>>> for( continuation parameter p = 0 to 1){
>>>> 
>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>> }
>>>> 
>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>> 
>>>  So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>> 
>>>  If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>> 
>>> Do not use -snes_grid_sequencing  
>>> 
>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>> 
>>> Call SNESSetGridSequence()
>>> 
>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150907/87cec701/attachment.html>

From knepley at gmail.com  Mon Sep  7 19:40:04 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 7 Sep 2015 20:40:04 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
Message-ID: <CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>

On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Barry,
>
> I finally got a chance to really try using the grid sequencing within my
> code.  I find that, in some cases, even if it can solve successfully on the
> coarsest mesh, the SNES fails, usually due to a line search failure, when
> it tries to compute along the grid sequence.  Would you have any
> suggestions?
>

I apologize if I have asked before, but can you give me -snes_view for the
solver? I could not find it in the email thread.

I would suggest trying to fiddle with the line search, or precondition it
with Richardson. It would be nice to see -snes_monitor
for the runs that fail, and then we can break down the residual into fields
and look at it again (if my custom residual monitor
does not work we can write one easily). Seeing which part of the residual
does not converge is key to designing the NASM
for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai,
present it. We need better monitoring in PETSc.

  Thanks,

    Matt


> -gideon
>
> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Yes, if i continue in this parameter on the coarse mesh, I can generally
> solve at all values. I do find that I need to do some amount of
> continuation to solve near the endpoint.  The problem is that on the coarse
> mesh, things are not fully resolved at all the values along the
> continuation parameter, and I would like to do refinement.
>
> One subtlety is that I actually want the intermediate continuation
> solutions  too.  Currently, without doing any grid sequence, I compute
> each, write it to disk, and then go on to the next one.  So I now need to
> go back an refine them.  I was thinking that perhaps I could refine them on
> the fly, dump them to disk, and use the coarse solution as the starting
> guess at the next iteration, but that would seem to require resetting the
> snes back to the coarse grid.
>
> The alternative would be to just script the mesh refinement in a post
> processing stage, where each value of the continuation is parameter is
> loaded on the coarse mesh, and refined.  Perhaps that?s the most practical
> thing to do.
>
>
>   I would do the following. Create your DM and create a SNES that will do
> the continuation
>
>   loop over continuation parameter
>
>        SNESSolve(snes,NULL,Ucoarse);
>
>        if (you decide you want to see the refined solution at this
> continuation point) {
>             SNESCreate(comm,&snesrefine);
>             SNESSetDM()
>             etc
>             SNESSetGridSequence(snesrefine,)
>             SNESSolve(snesrefine,0,Ucoarse);
>             SNESGetSolution(snesrefine,&Ufine);
>             VecView(Ufine or do whatever you want to do with the Ufine at
> that continuation point
>             SNESDestroy(snesrefine);
>       end if
>
>   end loop over continuation parameter.
>
>   Barry
>
>
> -gideon
>
> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>
> 3.  This problem is actually part of a continuation problem that roughly
> looks like this
>
> for( continuation parameter p = 0 to 1){
>
> solve with parameter p_i using solution from p_{i-1},
> }
>
> What I would like to do is to start the solver, for each value of
> parameter p_i on the coarse mesh, and then do grid sequencing on that.  But
> it appears that after doing grid sequencing on the initial p_0 = 0, the
> SNES is set to use the finer mesh.
>
>
>  So you are using continuation to give you a good enough initial guess on
> the coarse level to even get convergence on the coarse level? First I would
> check if you even need the continuation (or can you not even solve the
> coarse problem without it).
>
>  If you do need the continuation then you will need to tweak how you do
> the grid sequencing. I think this will work:
>
> Do not use -snes_grid_sequencing
>
> Run SNESSolve() as many times as you want with your continuation
> parameter. This will all happen on the coarse mesh.
>
> Call SNESSetGridSequence()
>
> Then call SNESSolve() again and it will do one solve on the coarse level
> and then interpolate to the next level etc.
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150907/88a7baf1/attachment.html>

From gideon.simpson at gmail.com  Mon Sep  7 19:49:59 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 7 Sep 2015 20:49:59 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
Message-ID: <FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>

No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.

      0 SNES Function norm 4.027481756921e-09 
      1 SNES Function norm 1.760477878365e-12 
    Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
    0 SNES Function norm 5.066222213176e+03 
    1 SNES Function norm 8.484697184230e+02 
    2 SNES Function norm 6.549559723294e+02 
    3 SNES Function norm 5.770723278153e+02 
    4 SNES Function norm 5.237702240594e+02 
    5 SNES Function norm 4.753909019848e+02 
    6 SNES Function norm 4.221784590755e+02 
    7 SNES Function norm 3.806525080483e+02 
    8 SNES Function norm 3.762054656019e+02 
    9 SNES Function norm 3.758975226873e+02 
   10 SNES Function norm 3.757032042706e+02 
   11 SNES Function norm 3.728798164234e+02 
   12 SNES Function norm 3.723078741075e+02 
   13 SNES Function norm 3.721848059825e+02 
   14 SNES Function norm 3.720227575629e+02 
   15 SNES Function norm 3.720051998555e+02 
   16 SNES Function norm 3.718945430587e+02 
   17 SNES Function norm 3.700412694044e+02 
   18 SNES Function norm 3.351964889461e+02 
   19 SNES Function norm 3.096016086233e+02 
   20 SNES Function norm 3.008410789787e+02 
   21 SNES Function norm 2.752316716557e+02 
   22 SNES Function norm 2.707658474165e+02 
   23 SNES Function norm 2.698436736049e+02 
   24 SNES Function norm 2.618233857172e+02 
   25 SNES Function norm 2.600121920634e+02 
   26 SNES Function norm 2.585046423168e+02 
   27 SNES Function norm 2.568551090220e+02 
   28 SNES Function norm 2.556404537064e+02 
   29 SNES Function norm 2.536353523683e+02 
   30 SNES Function norm 2.533596070171e+02 
   31 SNES Function norm 2.532324379596e+02 
   32 SNES Function norm 2.531842335211e+02 
   33 SNES Function norm 2.531684527520e+02 
   34 SNES Function norm 2.531637604618e+02 
   35 SNES Function norm 2.531624767821e+02 
   36 SNES Function norm 2.531621359093e+02 
   37 SNES Function norm 2.531620504925e+02 
   38 SNES Function norm 2.531620350055e+02 
   39 SNES Function norm 2.531620310522e+02 
   40 SNES Function norm 2.531620300471e+02 
   41 SNES Function norm 2.531620298084e+02 
   42 SNES Function norm 2.531620297478e+02 
   43 SNES Function norm 2.531620297324e+02 
   44 SNES Function norm 2.531620297303e+02 
   45 SNES Function norm 2.531620297302e+02 
  Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
  0 SNES Function norm 9.636339304380e+03 
  1 SNES Function norm 8.997731184634e+03 
  2 SNES Function norm 8.120498349232e+03 
  3 SNES Function norm 7.322379894820e+03 
  4 SNES Function norm 6.599581599149e+03 
  5 SNES Function norm 6.374872854688e+03 
  6 SNES Function norm 6.372518007653e+03 
  7 SNES Function norm 6.073996314301e+03 
  8 SNES Function norm 5.635965277054e+03 
  9 SNES Function norm 5.155389064046e+03 
 10 SNES Function norm 5.080567902638e+03 
 11 SNES Function norm 5.058878643969e+03 
 12 SNES Function norm 5.058835649793e+03 
 13 SNES Function norm 5.058491285707e+03 
 14 SNES Function norm 5.057452865337e+03 
 15 SNES Function norm 5.057226140688e+03 
 16 SNES Function norm 5.056651272898e+03 
 17 SNES Function norm 5.056575190057e+03 
 18 SNES Function norm 5.056574632598e+03 
 19 SNES Function norm 5.056574520229e+03 
 20 SNES Function norm 5.056574492569e+03 
 21 SNES Function norm 5.056574485124e+03 
 22 SNES Function norm 5.056574483029e+03 
 23 SNES Function norm 5.056574482427e+03 
 24 SNES Function norm 5.056574482302e+03 
 25 SNES Function norm 5.056574482287e+03 
 26 SNES Function norm 5.056574482282e+03 
 27 SNES Function norm 5.056574482281e+03 
Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
SNES Object: 1 MPI processes
  type: newtonls
  maximum iterations=50, maximum function evaluations=10000
  tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
  total number of linear solver iterations=28
  total number of function evaluations=323
  total number of grid sequence refinements=2
  SNESLineSearch Object:   1 MPI processes
    type: bt
      interpolation: cubic
      alpha=1.000000e-04
    maxstep=1.000000e+08, minlambda=1.000000e-12
    tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
    maximum iterations=40
  KSP Object:   1 MPI processes
    type: gmres
      GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
      GMRES: happy breakdown tolerance 1e-30
    maximum iterations=10000, initial guess is zero
    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
    left preconditioning
    using PRECONDITIONED norm type for convergence test
  PC Object:   1 MPI processes
    type: lu
      LU: out-of-place factorization
      tolerance for zero pivot 2.22045e-14
      matrix ordering: nd
      factor fill ratio given 0, needed 0
        Factored matrix follows:
          Mat Object:           1 MPI processes
            type: seqaij
            rows=15991, cols=15991
            package used to perform factorization: mumps
            total: nonzeros=255801, allocated nonzeros=255801
            total number of mallocs used during MatSetValues calls =0
              MUMPS run parameters:
                SYM (matrix type):                   0 
                PAR (host participation):            1 
                ICNTL(1) (output for error):         6 
                ICNTL(2) (output of diagnostic msg): 0 
                ICNTL(3) (output for global info):   0 
                ICNTL(4) (level of printing):        0 
                ICNTL(5) (input mat struct):         0 
                ICNTL(6) (matrix prescaling):        7 
                ICNTL(7) (sequentia matrix ordering):6 
                ICNTL(8) (scalling strategy):        77 
                ICNTL(10) (max num of refinements):  0 
                ICNTL(11) (error analysis):          0 
                ICNTL(12) (efficiency control):                         1 
                ICNTL(13) (efficiency control):                         0 
                ICNTL(14) (percentage of estimated workspace increase): 20 
                ICNTL(18) (input mat struct):                           0 
                ICNTL(19) (Shur complement info):                       0 
                ICNTL(20) (rhs sparse pattern):                         0 
                ICNTL(21) (somumpstion struct):                            0 
                ICNTL(22) (in-core/out-of-core facility):               0 
                ICNTL(23) (max size of memory can be allocated locally):0 
                ICNTL(24) (detection of null pivot rows):               0 
                ICNTL(25) (computation of a null space basis):          0 
                ICNTL(26) (Schur options for rhs or solution):          0 
                ICNTL(27) (experimental parameter):                     -8 
                ICNTL(28) (use parallel or sequential ordering):        1 
                ICNTL(29) (parallel ordering):                          0 
                ICNTL(30) (user-specified set of entries in inv(A)):    0 
                ICNTL(31) (factors is discarded in the solve phase):    0 
                ICNTL(33) (compute determinant):                        0 
                CNTL(1) (relative pivoting threshold):      0.01 
                CNTL(2) (stopping criterion of refinement): 1.49012e-08 
                CNTL(3) (absomumpste pivoting threshold):      0 
                CNTL(4) (vamumpse of static pivoting):         -1 
                CNTL(5) (fixation for null pivots):         0 
                RINFO(1) (local estimated flops for the elimination after analysis): 
                  [0] 1.95838e+06 
                RINFO(2) (local estimated flops for the assembly after factorization): 
                  [0]  143924 
                RINFO(3) (local estimated flops for the elimination after factorization): 
                  [0]  1.95943e+06 
                INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
                [0] 7 
                INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
                  [0] 7 
                INFO(23) (num of pivots eliminated on this processor after factorization): 
                  [0] 15991 
                RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
                RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
                RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
                (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
                INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
                INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
                INFOG(5) (estimated maximum front size in the complete tree): 11 
                INFOG(6) (number of nodes in the complete tree): 3996 
                INFOG(7) (ordering option effectively use after analysis): 6 
                INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
                INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
                INFOG(10) (total integer space store the matrix factors after factorization): 127890 
                INFOG(11) (order of largest frontal matrix after factorization): 11 
                INFOG(12) (number of off-diagonal pivots): 19 
                INFOG(13) (number of delayed pivots after factorization): 8 
                INFOG(14) (number of memory compress after factorization): 0 
                INFOG(15) (number of steps of iterative refinement after solution): 0 
                INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
                INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
                INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
                INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
                INFOG(20) (estimated number of entries in the factors): 255801 
                INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
                INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
                INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
                INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
                INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
                INFOG(28) (after factorization: number of null pivots encountered): 0
                INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
                INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
                INFOG(32) (after analysis: type of analysis done): 1
                INFOG(33) (value used for ICNTL(8)): 7
                INFOG(34) (exponent of the determinant if determinant is requested): 0
    linear system matrix = precond matrix:
    Mat Object:     1 MPI processes
      type: seqaij
      rows=15991, cols=15991
      total: nonzeros=223820, allocated nonzeros=431698
      total number of mallocs used during MatSetValues calls =15991
        using I-node routines: found 4000 nodes, limit used is 5




-gideon

> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> Barry,
> 
> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
> 
> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
> 
> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
> 
>   Thanks,
> 
>     Matt
>  
> -gideon
> 
>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>> 
>> 
>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>> 
>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>> 
>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>> 
>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>> 
>>   I would do the following. Create your DM and create a SNES that will do the continuation
>> 
>>   loop over continuation parameter
>> 
>>        SNESSolve(snes,NULL,Ucoarse);
>> 
>>        if (you decide you want to see the refined solution at this continuation point) {
>>             SNESCreate(comm,&snesrefine);
>>             SNESSetDM()
>>             etc
>>             SNESSetGridSequence(snesrefine,)
>>             SNESSolve(snesrefine,0,Ucoarse);
>>             SNESGetSolution(snesrefine,&Ufine);
>>             VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>             SNESDestroy(snesrefine);
>>       end if
>> 
>>   end loop over continuation parameter.
>> 
>>   Barry
>> 
>>> 
>>> -gideon
>>> 
>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>> 
>>>>> for( continuation parameter p = 0 to 1){
>>>>> 
>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>> }
>>>>> 
>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>> 
>>>>  So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>> 
>>>>  If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>> 
>>>> Do not use -snes_grid_sequencing  
>>>> 
>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>> 
>>>> Call SNESSetGridSequence()
>>>> 
>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>> 
>> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150907/c4851b23/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon Sep  7 20:01:48 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Sep 2015 20:01:48 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
Message-ID: <8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>


  My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
-snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?

  Barry



> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
> 
>       0 SNES Function norm 4.027481756921e-09 
>       1 SNES Function norm 1.760477878365e-12 
>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>     0 SNES Function norm 5.066222213176e+03 
>     1 SNES Function norm 8.484697184230e+02 
>     2 SNES Function norm 6.549559723294e+02 
>     3 SNES Function norm 5.770723278153e+02 
>     4 SNES Function norm 5.237702240594e+02 
>     5 SNES Function norm 4.753909019848e+02 
>     6 SNES Function norm 4.221784590755e+02 
>     7 SNES Function norm 3.806525080483e+02 
>     8 SNES Function norm 3.762054656019e+02 
>     9 SNES Function norm 3.758975226873e+02 
>    10 SNES Function norm 3.757032042706e+02 
>    11 SNES Function norm 3.728798164234e+02 
>    12 SNES Function norm 3.723078741075e+02 
>    13 SNES Function norm 3.721848059825e+02 
>    14 SNES Function norm 3.720227575629e+02 
>    15 SNES Function norm 3.720051998555e+02 
>    16 SNES Function norm 3.718945430587e+02 
>    17 SNES Function norm 3.700412694044e+02 
>    18 SNES Function norm 3.351964889461e+02 
>    19 SNES Function norm 3.096016086233e+02 
>    20 SNES Function norm 3.008410789787e+02 
>    21 SNES Function norm 2.752316716557e+02 
>    22 SNES Function norm 2.707658474165e+02 
>    23 SNES Function norm 2.698436736049e+02 
>    24 SNES Function norm 2.618233857172e+02 
>    25 SNES Function norm 2.600121920634e+02 
>    26 SNES Function norm 2.585046423168e+02 
>    27 SNES Function norm 2.568551090220e+02 
>    28 SNES Function norm 2.556404537064e+02 
>    29 SNES Function norm 2.536353523683e+02 
>    30 SNES Function norm 2.533596070171e+02 
>    31 SNES Function norm 2.532324379596e+02 
>    32 SNES Function norm 2.531842335211e+02 
>    33 SNES Function norm 2.531684527520e+02 
>    34 SNES Function norm 2.531637604618e+02 
>    35 SNES Function norm 2.531624767821e+02 
>    36 SNES Function norm 2.531621359093e+02 
>    37 SNES Function norm 2.531620504925e+02 
>    38 SNES Function norm 2.531620350055e+02 
>    39 SNES Function norm 2.531620310522e+02 
>    40 SNES Function norm 2.531620300471e+02 
>    41 SNES Function norm 2.531620298084e+02 
>    42 SNES Function norm 2.531620297478e+02 
>    43 SNES Function norm 2.531620297324e+02 
>    44 SNES Function norm 2.531620297303e+02 
>    45 SNES Function norm 2.531620297302e+02 
>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>   0 SNES Function norm 9.636339304380e+03 
>   1 SNES Function norm 8.997731184634e+03 
>   2 SNES Function norm 8.120498349232e+03 
>   3 SNES Function norm 7.322379894820e+03 
>   4 SNES Function norm 6.599581599149e+03 
>   5 SNES Function norm 6.374872854688e+03 
>   6 SNES Function norm 6.372518007653e+03 
>   7 SNES Function norm 6.073996314301e+03 
>   8 SNES Function norm 5.635965277054e+03 
>   9 SNES Function norm 5.155389064046e+03 
>  10 SNES Function norm 5.080567902638e+03 
>  11 SNES Function norm 5.058878643969e+03 
>  12 SNES Function norm 5.058835649793e+03 
>  13 SNES Function norm 5.058491285707e+03 
>  14 SNES Function norm 5.057452865337e+03 
>  15 SNES Function norm 5.057226140688e+03 
>  16 SNES Function norm 5.056651272898e+03 
>  17 SNES Function norm 5.056575190057e+03 
>  18 SNES Function norm 5.056574632598e+03 
>  19 SNES Function norm 5.056574520229e+03 
>  20 SNES Function norm 5.056574492569e+03 
>  21 SNES Function norm 5.056574485124e+03 
>  22 SNES Function norm 5.056574483029e+03 
>  23 SNES Function norm 5.056574482427e+03 
>  24 SNES Function norm 5.056574482302e+03 
>  25 SNES Function norm 5.056574482287e+03 
>  26 SNES Function norm 5.056574482282e+03 
>  27 SNES Function norm 5.056574482281e+03 
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
> SNES Object: 1 MPI processes
>   type: newtonls
>   maximum iterations=50, maximum function evaluations=10000
>   tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>   total number of linear solver iterations=28
>   total number of function evaluations=323
>   total number of grid sequence refinements=2
>   SNESLineSearch Object:   1 MPI processes
>     type: bt
>       interpolation: cubic
>       alpha=1.000000e-04
>     maxstep=1.000000e+08, minlambda=1.000000e-12
>     tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>     maximum iterations=40
>   KSP Object:   1 MPI processes
>     type: gmres
>       GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>       GMRES: happy breakdown tolerance 1e-30
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>     left preconditioning
>     using PRECONDITIONED norm type for convergence test
>   PC Object:   1 MPI processes
>     type: lu
>       LU: out-of-place factorization
>       tolerance for zero pivot 2.22045e-14
>       matrix ordering: nd
>       factor fill ratio given 0, needed 0
>         Factored matrix follows:
>           Mat Object:           1 MPI processes
>             type: seqaij
>             rows=15991, cols=15991
>             package used to perform factorization: mumps
>             total: nonzeros=255801, allocated nonzeros=255801
>             total number of mallocs used during MatSetValues calls =0
>               MUMPS run parameters:
>                 SYM (matrix type):                   0 
>                 PAR (host participation):            1 
>                 ICNTL(1) (output for error):         6 
>                 ICNTL(2) (output of diagnostic msg): 0 
>                 ICNTL(3) (output for global info):   0 
>                 ICNTL(4) (level of printing):        0 
>                 ICNTL(5) (input mat struct):         0 
>                 ICNTL(6) (matrix prescaling):        7 
>                 ICNTL(7) (sequentia matrix ordering):6 
>                 ICNTL(8) (scalling strategy):        77 
>                 ICNTL(10) (max num of refinements):  0 
>                 ICNTL(11) (error analysis):          0 
>                 ICNTL(12) (efficiency control):                         1 
>                 ICNTL(13) (efficiency control):                         0 
>                 ICNTL(14) (percentage of estimated workspace increase): 20 
>                 ICNTL(18) (input mat struct):                           0 
>                 ICNTL(19) (Shur complement info):                       0 
>                 ICNTL(20) (rhs sparse pattern):                         0 
>                 ICNTL(21) (somumpstion struct):                            0 
>                 ICNTL(22) (in-core/out-of-core facility):               0 
>                 ICNTL(23) (max size of memory can be allocated locally):0 
>                 ICNTL(24) (detection of null pivot rows):               0 
>                 ICNTL(25) (computation of a null space basis):          0 
>                 ICNTL(26) (Schur options for rhs or solution):          0 
>                 ICNTL(27) (experimental parameter):                     -8 
>                 ICNTL(28) (use parallel or sequential ordering):        1 
>                 ICNTL(29) (parallel ordering):                          0 
>                 ICNTL(30) (user-specified set of entries in inv(A)):    0 
>                 ICNTL(31) (factors is discarded in the solve phase):    0 
>                 ICNTL(33) (compute determinant):                        0 
>                 CNTL(1) (relative pivoting threshold):      0.01 
>                 CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>                 CNTL(3) (absomumpste pivoting threshold):      0 
>                 CNTL(4) (vamumpse of static pivoting):         -1 
>                 CNTL(5) (fixation for null pivots):         0 
>                 RINFO(1) (local estimated flops for the elimination after analysis): 
>                   [0] 1.95838e+06 
>                 RINFO(2) (local estimated flops for the assembly after factorization): 
>                   [0]  143924 
>                 RINFO(3) (local estimated flops for the elimination after factorization): 
>                   [0]  1.95943e+06 
>                 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>                 [0] 7 
>                 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>                   [0] 7 
>                 INFO(23) (num of pivots eliminated on this processor after factorization): 
>                   [0] 15991 
>                 RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>                 RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>                 RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>                 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>                 INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>                 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>                 INFOG(5) (estimated maximum front size in the complete tree): 11 
>                 INFOG(6) (number of nodes in the complete tree): 3996 
>                 INFOG(7) (ordering option effectively use after analysis): 6 
>                 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>                 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>                 INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>                 INFOG(11) (order of largest frontal matrix after factorization): 11 
>                 INFOG(12) (number of off-diagonal pivots): 19 
>                 INFOG(13) (number of delayed pivots after factorization): 8 
>                 INFOG(14) (number of memory compress after factorization): 0 
>                 INFOG(15) (number of steps of iterative refinement after solution): 0 
>                 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>                 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>                 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>                 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>                 INFOG(20) (estimated number of entries in the factors): 255801 
>                 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>                 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>                 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>                 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>                 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>                 INFOG(28) (after factorization: number of null pivots encountered): 0
>                 INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>                 INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>                 INFOG(32) (after analysis: type of analysis done): 1
>                 INFOG(33) (value used for ICNTL(8)): 7
>                 INFOG(34) (exponent of the determinant if determinant is requested): 0
>     linear system matrix = precond matrix:
>     Mat Object:     1 MPI processes
>       type: seqaij
>       rows=15991, cols=15991
>       total: nonzeros=223820, allocated nonzeros=431698
>       total number of mallocs used during MatSetValues calls =15991
>         using I-node routines: found 4000 nodes, limit used is 5
> 
> 
> 
> 
> -gideon
> 
>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> Barry,
>> 
>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>> 
>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>> 
>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>> 
>>   Thanks,
>> 
>>     Matt
>>  
>> -gideon
>> 
>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>> 
>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>> 
>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>> 
>>>   I would do the following. Create your DM and create a SNES that will do the continuation
>>> 
>>>   loop over continuation parameter
>>> 
>>>        SNESSolve(snes,NULL,Ucoarse);
>>> 
>>>        if (you decide you want to see the refined solution at this continuation point) {
>>>             SNESCreate(comm,&snesrefine);
>>>             SNESSetDM()
>>>             etc
>>>             SNESSetGridSequence(snesrefine,)
>>>             SNESSolve(snesrefine,0,Ucoarse);
>>>             SNESGetSolution(snesrefine,&Ufine);
>>>             VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>             SNESDestroy(snesrefine);
>>>       end if
>>> 
>>>   end loop over continuation parameter.
>>> 
>>>   Barry
>>> 
>>>> 
>>>> -gideon
>>>> 
>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>> 
>>>>>> for( continuation parameter p = 0 to 1){
>>>>>> 
>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>> }
>>>>>> 
>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>> 
>>>>>  So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>> 
>>>>>  If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>> 
>>>>> Do not use -snes_grid_sequencing  
>>>>> 
>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>> 
>>>>> Call SNESSetGridSequence()
>>>>> 
>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>> 
>>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 


From gideon.simpson at gmail.com  Mon Sep  7 20:09:01 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 7 Sep 2015 21:09:01 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
Message-ID: <C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>

I?m getting an error with -snes_mf_operator, 

  0 SNES Function norm 1.421454390131e-02 
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
[0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
[0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
[0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

-gideon

> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
> 
>  Barry
> 
> 
> 
>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>> 
>>      0 SNES Function norm 4.027481756921e-09 
>>      1 SNES Function norm 1.760477878365e-12 
>>    Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>    0 SNES Function norm 5.066222213176e+03 
>>    1 SNES Function norm 8.484697184230e+02 
>>    2 SNES Function norm 6.549559723294e+02 
>>    3 SNES Function norm 5.770723278153e+02 
>>    4 SNES Function norm 5.237702240594e+02 
>>    5 SNES Function norm 4.753909019848e+02 
>>    6 SNES Function norm 4.221784590755e+02 
>>    7 SNES Function norm 3.806525080483e+02 
>>    8 SNES Function norm 3.762054656019e+02 
>>    9 SNES Function norm 3.758975226873e+02 
>>   10 SNES Function norm 3.757032042706e+02 
>>   11 SNES Function norm 3.728798164234e+02 
>>   12 SNES Function norm 3.723078741075e+02 
>>   13 SNES Function norm 3.721848059825e+02 
>>   14 SNES Function norm 3.720227575629e+02 
>>   15 SNES Function norm 3.720051998555e+02 
>>   16 SNES Function norm 3.718945430587e+02 
>>   17 SNES Function norm 3.700412694044e+02 
>>   18 SNES Function norm 3.351964889461e+02 
>>   19 SNES Function norm 3.096016086233e+02 
>>   20 SNES Function norm 3.008410789787e+02 
>>   21 SNES Function norm 2.752316716557e+02 
>>   22 SNES Function norm 2.707658474165e+02 
>>   23 SNES Function norm 2.698436736049e+02 
>>   24 SNES Function norm 2.618233857172e+02 
>>   25 SNES Function norm 2.600121920634e+02 
>>   26 SNES Function norm 2.585046423168e+02 
>>   27 SNES Function norm 2.568551090220e+02 
>>   28 SNES Function norm 2.556404537064e+02 
>>   29 SNES Function norm 2.536353523683e+02 
>>   30 SNES Function norm 2.533596070171e+02 
>>   31 SNES Function norm 2.532324379596e+02 
>>   32 SNES Function norm 2.531842335211e+02 
>>   33 SNES Function norm 2.531684527520e+02 
>>   34 SNES Function norm 2.531637604618e+02 
>>   35 SNES Function norm 2.531624767821e+02 
>>   36 SNES Function norm 2.531621359093e+02 
>>   37 SNES Function norm 2.531620504925e+02 
>>   38 SNES Function norm 2.531620350055e+02 
>>   39 SNES Function norm 2.531620310522e+02 
>>   40 SNES Function norm 2.531620300471e+02 
>>   41 SNES Function norm 2.531620298084e+02 
>>   42 SNES Function norm 2.531620297478e+02 
>>   43 SNES Function norm 2.531620297324e+02 
>>   44 SNES Function norm 2.531620297303e+02 
>>   45 SNES Function norm 2.531620297302e+02 
>>  Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>  0 SNES Function norm 9.636339304380e+03 
>>  1 SNES Function norm 8.997731184634e+03 
>>  2 SNES Function norm 8.120498349232e+03 
>>  3 SNES Function norm 7.322379894820e+03 
>>  4 SNES Function norm 6.599581599149e+03 
>>  5 SNES Function norm 6.374872854688e+03 
>>  6 SNES Function norm 6.372518007653e+03 
>>  7 SNES Function norm 6.073996314301e+03 
>>  8 SNES Function norm 5.635965277054e+03 
>>  9 SNES Function norm 5.155389064046e+03 
>> 10 SNES Function norm 5.080567902638e+03 
>> 11 SNES Function norm 5.058878643969e+03 
>> 12 SNES Function norm 5.058835649793e+03 
>> 13 SNES Function norm 5.058491285707e+03 
>> 14 SNES Function norm 5.057452865337e+03 
>> 15 SNES Function norm 5.057226140688e+03 
>> 16 SNES Function norm 5.056651272898e+03 
>> 17 SNES Function norm 5.056575190057e+03 
>> 18 SNES Function norm 5.056574632598e+03 
>> 19 SNES Function norm 5.056574520229e+03 
>> 20 SNES Function norm 5.056574492569e+03 
>> 21 SNES Function norm 5.056574485124e+03 
>> 22 SNES Function norm 5.056574483029e+03 
>> 23 SNES Function norm 5.056574482427e+03 
>> 24 SNES Function norm 5.056574482302e+03 
>> 25 SNES Function norm 5.056574482287e+03 
>> 26 SNES Function norm 5.056574482282e+03 
>> 27 SNES Function norm 5.056574482281e+03 
>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>> SNES Object: 1 MPI processes
>>  type: newtonls
>>  maximum iterations=50, maximum function evaluations=10000
>>  tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>  total number of linear solver iterations=28
>>  total number of function evaluations=323
>>  total number of grid sequence refinements=2
>>  SNESLineSearch Object:   1 MPI processes
>>    type: bt
>>      interpolation: cubic
>>      alpha=1.000000e-04
>>    maxstep=1.000000e+08, minlambda=1.000000e-12
>>    tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>    maximum iterations=40
>>  KSP Object:   1 MPI processes
>>    type: gmres
>>      GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>      GMRES: happy breakdown tolerance 1e-30
>>    maximum iterations=10000, initial guess is zero
>>    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>    left preconditioning
>>    using PRECONDITIONED norm type for convergence test
>>  PC Object:   1 MPI processes
>>    type: lu
>>      LU: out-of-place factorization
>>      tolerance for zero pivot 2.22045e-14
>>      matrix ordering: nd
>>      factor fill ratio given 0, needed 0
>>        Factored matrix follows:
>>          Mat Object:           1 MPI processes
>>            type: seqaij
>>            rows=15991, cols=15991
>>            package used to perform factorization: mumps
>>            total: nonzeros=255801, allocated nonzeros=255801
>>            total number of mallocs used during MatSetValues calls =0
>>              MUMPS run parameters:
>>                SYM (matrix type):                   0 
>>                PAR (host participation):            1 
>>                ICNTL(1) (output for error):         6 
>>                ICNTL(2) (output of diagnostic msg): 0 
>>                ICNTL(3) (output for global info):   0 
>>                ICNTL(4) (level of printing):        0 
>>                ICNTL(5) (input mat struct):         0 
>>                ICNTL(6) (matrix prescaling):        7 
>>                ICNTL(7) (sequentia matrix ordering):6 
>>                ICNTL(8) (scalling strategy):        77 
>>                ICNTL(10) (max num of refinements):  0 
>>                ICNTL(11) (error analysis):          0 
>>                ICNTL(12) (efficiency control):                         1 
>>                ICNTL(13) (efficiency control):                         0 
>>                ICNTL(14) (percentage of estimated workspace increase): 20 
>>                ICNTL(18) (input mat struct):                           0 
>>                ICNTL(19) (Shur complement info):                       0 
>>                ICNTL(20) (rhs sparse pattern):                         0 
>>                ICNTL(21) (somumpstion struct):                            0 
>>                ICNTL(22) (in-core/out-of-core facility):               0 
>>                ICNTL(23) (max size of memory can be allocated locally):0 
>>                ICNTL(24) (detection of null pivot rows):               0 
>>                ICNTL(25) (computation of a null space basis):          0 
>>                ICNTL(26) (Schur options for rhs or solution):          0 
>>                ICNTL(27) (experimental parameter):                     -8 
>>                ICNTL(28) (use parallel or sequential ordering):        1 
>>                ICNTL(29) (parallel ordering):                          0 
>>                ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>                ICNTL(31) (factors is discarded in the solve phase):    0 
>>                ICNTL(33) (compute determinant):                        0 
>>                CNTL(1) (relative pivoting threshold):      0.01 
>>                CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>                CNTL(3) (absomumpste pivoting threshold):      0 
>>                CNTL(4) (vamumpse of static pivoting):         -1 
>>                CNTL(5) (fixation for null pivots):         0 
>>                RINFO(1) (local estimated flops for the elimination after analysis): 
>>                  [0] 1.95838e+06 
>>                RINFO(2) (local estimated flops for the assembly after factorization): 
>>                  [0]  143924 
>>                RINFO(3) (local estimated flops for the elimination after factorization): 
>>                  [0]  1.95943e+06 
>>                INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>                [0] 7 
>>                INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>                  [0] 7 
>>                INFO(23) (num of pivots eliminated on this processor after factorization): 
>>                  [0] 15991 
>>                RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>                RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>                RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>                (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>                INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>                INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>                INFOG(5) (estimated maximum front size in the complete tree): 11 
>>                INFOG(6) (number of nodes in the complete tree): 3996 
>>                INFOG(7) (ordering option effectively use after analysis): 6 
>>                INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>                INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>                INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>                INFOG(11) (order of largest frontal matrix after factorization): 11 
>>                INFOG(12) (number of off-diagonal pivots): 19 
>>                INFOG(13) (number of delayed pivots after factorization): 8 
>>                INFOG(14) (number of memory compress after factorization): 0 
>>                INFOG(15) (number of steps of iterative refinement after solution): 0 
>>                INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>                INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>                INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>                INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>                INFOG(20) (estimated number of entries in the factors): 255801 
>>                INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>                INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>                INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>                INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>                INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>                INFOG(28) (after factorization: number of null pivots encountered): 0
>>                INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>                INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>                INFOG(32) (after analysis: type of analysis done): 1
>>                INFOG(33) (value used for ICNTL(8)): 7
>>                INFOG(34) (exponent of the determinant if determinant is requested): 0
>>    linear system matrix = precond matrix:
>>    Mat Object:     1 MPI processes
>>      type: seqaij
>>      rows=15991, cols=15991
>>      total: nonzeros=223820, allocated nonzeros=431698
>>      total number of mallocs used during MatSetValues calls =15991
>>        using I-node routines: found 4000 nodes, limit used is 5
>> 
>> 
>> 
>> 
>> -gideon
>> 
>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> Barry,
>>> 
>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>> 
>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>> 
>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>> 
>>>  Thanks,
>>> 
>>>    Matt
>>> 
>>> -gideon
>>> 
>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>> 
>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>> 
>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>> 
>>>>  I would do the following. Create your DM and create a SNES that will do the continuation
>>>> 
>>>>  loop over continuation parameter
>>>> 
>>>>       SNESSolve(snes,NULL,Ucoarse);
>>>> 
>>>>       if (you decide you want to see the refined solution at this continuation point) {
>>>>            SNESCreate(comm,&snesrefine);
>>>>            SNESSetDM()
>>>>            etc
>>>>            SNESSetGridSequence(snesrefine,)
>>>>            SNESSolve(snesrefine,0,Ucoarse);
>>>>            SNESGetSolution(snesrefine,&Ufine);
>>>>            VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>            SNESDestroy(snesrefine);
>>>>      end if
>>>> 
>>>>  end loop over continuation parameter.
>>>> 
>>>>  Barry
>>>> 
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>> 
>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>> 
>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>> }
>>>>>>> 
>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>> 
>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>> 
>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>> 
>>>>>> Do not use -snes_grid_sequencing  
>>>>>> 
>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>> 
>>>>>> Call SNESSetGridSequence()
>>>>>> 
>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150907/65f1544c/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon Sep  7 20:11:42 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Sep 2015 20:11:42 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
Message-ID: <EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>


  This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output



  Barry


> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I?m getting an error with -snes_mf_operator, 
> 
>   0 SNES Function norm 1.421454390131e-02 
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> 
> -gideon
> 
>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>> 
>>  Barry
>> 
>> 
>> 
>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>> 
>>>      0 SNES Function norm 4.027481756921e-09 
>>>      1 SNES Function norm 1.760477878365e-12 
>>>    Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>    0 SNES Function norm 5.066222213176e+03 
>>>    1 SNES Function norm 8.484697184230e+02 
>>>    2 SNES Function norm 6.549559723294e+02 
>>>    3 SNES Function norm 5.770723278153e+02 
>>>    4 SNES Function norm 5.237702240594e+02 
>>>    5 SNES Function norm 4.753909019848e+02 
>>>    6 SNES Function norm 4.221784590755e+02 
>>>    7 SNES Function norm 3.806525080483e+02 
>>>    8 SNES Function norm 3.762054656019e+02 
>>>    9 SNES Function norm 3.758975226873e+02 
>>>   10 SNES Function norm 3.757032042706e+02 
>>>   11 SNES Function norm 3.728798164234e+02 
>>>   12 SNES Function norm 3.723078741075e+02 
>>>   13 SNES Function norm 3.721848059825e+02 
>>>   14 SNES Function norm 3.720227575629e+02 
>>>   15 SNES Function norm 3.720051998555e+02 
>>>   16 SNES Function norm 3.718945430587e+02 
>>>   17 SNES Function norm 3.700412694044e+02 
>>>   18 SNES Function norm 3.351964889461e+02 
>>>   19 SNES Function norm 3.096016086233e+02 
>>>   20 SNES Function norm 3.008410789787e+02 
>>>   21 SNES Function norm 2.752316716557e+02 
>>>   22 SNES Function norm 2.707658474165e+02 
>>>   23 SNES Function norm 2.698436736049e+02 
>>>   24 SNES Function norm 2.618233857172e+02 
>>>   25 SNES Function norm 2.600121920634e+02 
>>>   26 SNES Function norm 2.585046423168e+02 
>>>   27 SNES Function norm 2.568551090220e+02 
>>>   28 SNES Function norm 2.556404537064e+02 
>>>   29 SNES Function norm 2.536353523683e+02 
>>>   30 SNES Function norm 2.533596070171e+02 
>>>   31 SNES Function norm 2.532324379596e+02 
>>>   32 SNES Function norm 2.531842335211e+02 
>>>   33 SNES Function norm 2.531684527520e+02 
>>>   34 SNES Function norm 2.531637604618e+02 
>>>   35 SNES Function norm 2.531624767821e+02 
>>>   36 SNES Function norm 2.531621359093e+02 
>>>   37 SNES Function norm 2.531620504925e+02 
>>>   38 SNES Function norm 2.531620350055e+02 
>>>   39 SNES Function norm 2.531620310522e+02 
>>>   40 SNES Function norm 2.531620300471e+02 
>>>   41 SNES Function norm 2.531620298084e+02 
>>>   42 SNES Function norm 2.531620297478e+02 
>>>   43 SNES Function norm 2.531620297324e+02 
>>>   44 SNES Function norm 2.531620297303e+02 
>>>   45 SNES Function norm 2.531620297302e+02 
>>>  Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>  0 SNES Function norm 9.636339304380e+03 
>>>  1 SNES Function norm 8.997731184634e+03 
>>>  2 SNES Function norm 8.120498349232e+03 
>>>  3 SNES Function norm 7.322379894820e+03 
>>>  4 SNES Function norm 6.599581599149e+03 
>>>  5 SNES Function norm 6.374872854688e+03 
>>>  6 SNES Function norm 6.372518007653e+03 
>>>  7 SNES Function norm 6.073996314301e+03 
>>>  8 SNES Function norm 5.635965277054e+03 
>>>  9 SNES Function norm 5.155389064046e+03 
>>> 10 SNES Function norm 5.080567902638e+03 
>>> 11 SNES Function norm 5.058878643969e+03 
>>> 12 SNES Function norm 5.058835649793e+03 
>>> 13 SNES Function norm 5.058491285707e+03 
>>> 14 SNES Function norm 5.057452865337e+03 
>>> 15 SNES Function norm 5.057226140688e+03 
>>> 16 SNES Function norm 5.056651272898e+03 
>>> 17 SNES Function norm 5.056575190057e+03 
>>> 18 SNES Function norm 5.056574632598e+03 
>>> 19 SNES Function norm 5.056574520229e+03 
>>> 20 SNES Function norm 5.056574492569e+03 
>>> 21 SNES Function norm 5.056574485124e+03 
>>> 22 SNES Function norm 5.056574483029e+03 
>>> 23 SNES Function norm 5.056574482427e+03 
>>> 24 SNES Function norm 5.056574482302e+03 
>>> 25 SNES Function norm 5.056574482287e+03 
>>> 26 SNES Function norm 5.056574482282e+03 
>>> 27 SNES Function norm 5.056574482281e+03 
>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>> SNES Object: 1 MPI processes
>>>  type: newtonls
>>>  maximum iterations=50, maximum function evaluations=10000
>>>  tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>  total number of linear solver iterations=28
>>>  total number of function evaluations=323
>>>  total number of grid sequence refinements=2
>>>  SNESLineSearch Object:   1 MPI processes
>>>    type: bt
>>>      interpolation: cubic
>>>      alpha=1.000000e-04
>>>    maxstep=1.000000e+08, minlambda=1.000000e-12
>>>    tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>    maximum iterations=40
>>>  KSP Object:   1 MPI processes
>>>    type: gmres
>>>      GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>      GMRES: happy breakdown tolerance 1e-30
>>>    maximum iterations=10000, initial guess is zero
>>>    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>    left preconditioning
>>>    using PRECONDITIONED norm type for convergence test
>>>  PC Object:   1 MPI processes
>>>    type: lu
>>>      LU: out-of-place factorization
>>>      tolerance for zero pivot 2.22045e-14
>>>      matrix ordering: nd
>>>      factor fill ratio given 0, needed 0
>>>        Factored matrix follows:
>>>          Mat Object:           1 MPI processes
>>>            type: seqaij
>>>            rows=15991, cols=15991
>>>            package used to perform factorization: mumps
>>>            total: nonzeros=255801, allocated nonzeros=255801
>>>            total number of mallocs used during MatSetValues calls =0
>>>              MUMPS run parameters:
>>>                SYM (matrix type):                   0 
>>>                PAR (host participation):            1 
>>>                ICNTL(1) (output for error):         6 
>>>                ICNTL(2) (output of diagnostic msg): 0 
>>>                ICNTL(3) (output for global info):   0 
>>>                ICNTL(4) (level of printing):        0 
>>>                ICNTL(5) (input mat struct):         0 
>>>                ICNTL(6) (matrix prescaling):        7 
>>>                ICNTL(7) (sequentia matrix ordering):6 
>>>                ICNTL(8) (scalling strategy):        77 
>>>                ICNTL(10) (max num of refinements):  0 
>>>                ICNTL(11) (error analysis):          0 
>>>                ICNTL(12) (efficiency control):                         1 
>>>                ICNTL(13) (efficiency control):                         0 
>>>                ICNTL(14) (percentage of estimated workspace increase): 20 
>>>                ICNTL(18) (input mat struct):                           0 
>>>                ICNTL(19) (Shur complement info):                       0 
>>>                ICNTL(20) (rhs sparse pattern):                         0 
>>>                ICNTL(21) (somumpstion struct):                            0 
>>>                ICNTL(22) (in-core/out-of-core facility):               0 
>>>                ICNTL(23) (max size of memory can be allocated locally):0 
>>>                ICNTL(24) (detection of null pivot rows):               0 
>>>                ICNTL(25) (computation of a null space basis):          0 
>>>                ICNTL(26) (Schur options for rhs or solution):          0 
>>>                ICNTL(27) (experimental parameter):                     -8 
>>>                ICNTL(28) (use parallel or sequential ordering):        1 
>>>                ICNTL(29) (parallel ordering):                          0 
>>>                ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>                ICNTL(31) (factors is discarded in the solve phase):    0 
>>>                ICNTL(33) (compute determinant):                        0 
>>>                CNTL(1) (relative pivoting threshold):      0.01 
>>>                CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>                CNTL(3) (absomumpste pivoting threshold):      0 
>>>                CNTL(4) (vamumpse of static pivoting):         -1 
>>>                CNTL(5) (fixation for null pivots):         0 
>>>                RINFO(1) (local estimated flops for the elimination after analysis): 
>>>                  [0] 1.95838e+06 
>>>                RINFO(2) (local estimated flops for the assembly after factorization): 
>>>                  [0]  143924 
>>>                RINFO(3) (local estimated flops for the elimination after factorization): 
>>>                  [0]  1.95943e+06 
>>>                INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>                [0] 7 
>>>                INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>                  [0] 7 
>>>                INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>                  [0] 15991 
>>>                RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>                RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>                RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>                (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>                INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>                INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>                INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>                INFOG(6) (number of nodes in the complete tree): 3996 
>>>                INFOG(7) (ordering option effectively use after analysis): 6 
>>>                INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>                INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>                INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>                INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>                INFOG(12) (number of off-diagonal pivots): 19 
>>>                INFOG(13) (number of delayed pivots after factorization): 8 
>>>                INFOG(14) (number of memory compress after factorization): 0 
>>>                INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>                INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>                INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>                INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>                INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>                INFOG(20) (estimated number of entries in the factors): 255801 
>>>                INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>                INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>                INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>                INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>                INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>                INFOG(28) (after factorization: number of null pivots encountered): 0
>>>                INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>                INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>                INFOG(32) (after analysis: type of analysis done): 1
>>>                INFOG(33) (value used for ICNTL(8)): 7
>>>                INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>    linear system matrix = precond matrix:
>>>    Mat Object:     1 MPI processes
>>>      type: seqaij
>>>      rows=15991, cols=15991
>>>      total: nonzeros=223820, allocated nonzeros=431698
>>>      total number of mallocs used during MatSetValues calls =15991
>>>        using I-node routines: found 4000 nodes, limit used is 5
>>> 
>>> 
>>> 
>>> 
>>> -gideon
>>> 
>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>> 
>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> Barry,
>>>> 
>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>> 
>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>> 
>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>> 
>>>>  Thanks,
>>>> 
>>>>    Matt
>>>> 
>>>> -gideon
>>>> 
>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>> 
>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>> 
>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>> 
>>>>>  I would do the following. Create your DM and create a SNES that will do the continuation
>>>>> 
>>>>>  loop over continuation parameter
>>>>> 
>>>>>       SNESSolve(snes,NULL,Ucoarse);
>>>>> 
>>>>>       if (you decide you want to see the refined solution at this continuation point) {
>>>>>            SNESCreate(comm,&snesrefine);
>>>>>            SNESSetDM()
>>>>>            etc
>>>>>            SNESSetGridSequence(snesrefine,)
>>>>>            SNESSolve(snesrefine,0,Ucoarse);
>>>>>            SNESGetSolution(snesrefine,&Ufine);
>>>>>            VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>            SNESDestroy(snesrefine);
>>>>>      end if
>>>>> 
>>>>>  end loop over continuation parameter.
>>>>> 
>>>>>  Barry
>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>> 
>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>> 
>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>> }
>>>>>>>> 
>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>> 
>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>> 
>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>> 
>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>> 
>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>> 
>>>>>>> Call SNESSetGridSequence()
>>>>>>> 
>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>> 
> 


From gideon.simpson at gmail.com  Mon Sep  7 20:18:35 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 7 Sep 2015 21:18:35 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
Message-ID: <09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>

Running with that flag gives me this:

[0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
Unable to start debugger: No such file or directory



-gideon

> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
> 
> 
> 
>  Barry
> 
> 
>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> I?m getting an error with -snes_mf_operator, 
>> 
>>  0 SNES Function norm 1.421454390131e-02 
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>> [0]PETSC ERROR: to get more information on the crash.
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Signal received
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>> 
>> -gideon
>> 
>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>> 
>>> Barry
>>> 
>>> 
>>> 
>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>> 
>>>>     0 SNES Function norm 4.027481756921e-09 
>>>>     1 SNES Function norm 1.760477878365e-12 
>>>>   Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>   0 SNES Function norm 5.066222213176e+03 
>>>>   1 SNES Function norm 8.484697184230e+02 
>>>>   2 SNES Function norm 6.549559723294e+02 
>>>>   3 SNES Function norm 5.770723278153e+02 
>>>>   4 SNES Function norm 5.237702240594e+02 
>>>>   5 SNES Function norm 4.753909019848e+02 
>>>>   6 SNES Function norm 4.221784590755e+02 
>>>>   7 SNES Function norm 3.806525080483e+02 
>>>>   8 SNES Function norm 3.762054656019e+02 
>>>>   9 SNES Function norm 3.758975226873e+02 
>>>>  10 SNES Function norm 3.757032042706e+02 
>>>>  11 SNES Function norm 3.728798164234e+02 
>>>>  12 SNES Function norm 3.723078741075e+02 
>>>>  13 SNES Function norm 3.721848059825e+02 
>>>>  14 SNES Function norm 3.720227575629e+02 
>>>>  15 SNES Function norm 3.720051998555e+02 
>>>>  16 SNES Function norm 3.718945430587e+02 
>>>>  17 SNES Function norm 3.700412694044e+02 
>>>>  18 SNES Function norm 3.351964889461e+02 
>>>>  19 SNES Function norm 3.096016086233e+02 
>>>>  20 SNES Function norm 3.008410789787e+02 
>>>>  21 SNES Function norm 2.752316716557e+02 
>>>>  22 SNES Function norm 2.707658474165e+02 
>>>>  23 SNES Function norm 2.698436736049e+02 
>>>>  24 SNES Function norm 2.618233857172e+02 
>>>>  25 SNES Function norm 2.600121920634e+02 
>>>>  26 SNES Function norm 2.585046423168e+02 
>>>>  27 SNES Function norm 2.568551090220e+02 
>>>>  28 SNES Function norm 2.556404537064e+02 
>>>>  29 SNES Function norm 2.536353523683e+02 
>>>>  30 SNES Function norm 2.533596070171e+02 
>>>>  31 SNES Function norm 2.532324379596e+02 
>>>>  32 SNES Function norm 2.531842335211e+02 
>>>>  33 SNES Function norm 2.531684527520e+02 
>>>>  34 SNES Function norm 2.531637604618e+02 
>>>>  35 SNES Function norm 2.531624767821e+02 
>>>>  36 SNES Function norm 2.531621359093e+02 
>>>>  37 SNES Function norm 2.531620504925e+02 
>>>>  38 SNES Function norm 2.531620350055e+02 
>>>>  39 SNES Function norm 2.531620310522e+02 
>>>>  40 SNES Function norm 2.531620300471e+02 
>>>>  41 SNES Function norm 2.531620298084e+02 
>>>>  42 SNES Function norm 2.531620297478e+02 
>>>>  43 SNES Function norm 2.531620297324e+02 
>>>>  44 SNES Function norm 2.531620297303e+02 
>>>>  45 SNES Function norm 2.531620297302e+02 
>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>> 0 SNES Function norm 9.636339304380e+03 
>>>> 1 SNES Function norm 8.997731184634e+03 
>>>> 2 SNES Function norm 8.120498349232e+03 
>>>> 3 SNES Function norm 7.322379894820e+03 
>>>> 4 SNES Function norm 6.599581599149e+03 
>>>> 5 SNES Function norm 6.374872854688e+03 
>>>> 6 SNES Function norm 6.372518007653e+03 
>>>> 7 SNES Function norm 6.073996314301e+03 
>>>> 8 SNES Function norm 5.635965277054e+03 
>>>> 9 SNES Function norm 5.155389064046e+03 
>>>> 10 SNES Function norm 5.080567902638e+03 
>>>> 11 SNES Function norm 5.058878643969e+03 
>>>> 12 SNES Function norm 5.058835649793e+03 
>>>> 13 SNES Function norm 5.058491285707e+03 
>>>> 14 SNES Function norm 5.057452865337e+03 
>>>> 15 SNES Function norm 5.057226140688e+03 
>>>> 16 SNES Function norm 5.056651272898e+03 
>>>> 17 SNES Function norm 5.056575190057e+03 
>>>> 18 SNES Function norm 5.056574632598e+03 
>>>> 19 SNES Function norm 5.056574520229e+03 
>>>> 20 SNES Function norm 5.056574492569e+03 
>>>> 21 SNES Function norm 5.056574485124e+03 
>>>> 22 SNES Function norm 5.056574483029e+03 
>>>> 23 SNES Function norm 5.056574482427e+03 
>>>> 24 SNES Function norm 5.056574482302e+03 
>>>> 25 SNES Function norm 5.056574482287e+03 
>>>> 26 SNES Function norm 5.056574482282e+03 
>>>> 27 SNES Function norm 5.056574482281e+03 
>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>> SNES Object: 1 MPI processes
>>>> type: newtonls
>>>> maximum iterations=50, maximum function evaluations=10000
>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>> total number of linear solver iterations=28
>>>> total number of function evaluations=323
>>>> total number of grid sequence refinements=2
>>>> SNESLineSearch Object:   1 MPI processes
>>>>   type: bt
>>>>     interpolation: cubic
>>>>     alpha=1.000000e-04
>>>>   maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>   tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>   maximum iterations=40
>>>> KSP Object:   1 MPI processes
>>>>   type: gmres
>>>>     GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>     GMRES: happy breakdown tolerance 1e-30
>>>>   maximum iterations=10000, initial guess is zero
>>>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>   left preconditioning
>>>>   using PRECONDITIONED norm type for convergence test
>>>> PC Object:   1 MPI processes
>>>>   type: lu
>>>>     LU: out-of-place factorization
>>>>     tolerance for zero pivot 2.22045e-14
>>>>     matrix ordering: nd
>>>>     factor fill ratio given 0, needed 0
>>>>       Factored matrix follows:
>>>>         Mat Object:           1 MPI processes
>>>>           type: seqaij
>>>>           rows=15991, cols=15991
>>>>           package used to perform factorization: mumps
>>>>           total: nonzeros=255801, allocated nonzeros=255801
>>>>           total number of mallocs used during MatSetValues calls =0
>>>>             MUMPS run parameters:
>>>>               SYM (matrix type):                   0 
>>>>               PAR (host participation):            1 
>>>>               ICNTL(1) (output for error):         6 
>>>>               ICNTL(2) (output of diagnostic msg): 0 
>>>>               ICNTL(3) (output for global info):   0 
>>>>               ICNTL(4) (level of printing):        0 
>>>>               ICNTL(5) (input mat struct):         0 
>>>>               ICNTL(6) (matrix prescaling):        7 
>>>>               ICNTL(7) (sequentia matrix ordering):6 
>>>>               ICNTL(8) (scalling strategy):        77 
>>>>               ICNTL(10) (max num of refinements):  0 
>>>>               ICNTL(11) (error analysis):          0 
>>>>               ICNTL(12) (efficiency control):                         1 
>>>>               ICNTL(13) (efficiency control):                         0 
>>>>               ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>               ICNTL(18) (input mat struct):                           0 
>>>>               ICNTL(19) (Shur complement info):                       0 
>>>>               ICNTL(20) (rhs sparse pattern):                         0 
>>>>               ICNTL(21) (somumpstion struct):                            0 
>>>>               ICNTL(22) (in-core/out-of-core facility):               0 
>>>>               ICNTL(23) (max size of memory can be allocated locally):0 
>>>>               ICNTL(24) (detection of null pivot rows):               0 
>>>>               ICNTL(25) (computation of a null space basis):          0 
>>>>               ICNTL(26) (Schur options for rhs or solution):          0 
>>>>               ICNTL(27) (experimental parameter):                     -8 
>>>>               ICNTL(28) (use parallel or sequential ordering):        1 
>>>>               ICNTL(29) (parallel ordering):                          0 
>>>>               ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>               ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>               ICNTL(33) (compute determinant):                        0 
>>>>               CNTL(1) (relative pivoting threshold):      0.01 
>>>>               CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>               CNTL(3) (absomumpste pivoting threshold):      0 
>>>>               CNTL(4) (vamumpse of static pivoting):         -1 
>>>>               CNTL(5) (fixation for null pivots):         0 
>>>>               RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>                 [0] 1.95838e+06 
>>>>               RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>                 [0]  143924 
>>>>               RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>                 [0]  1.95943e+06 
>>>>               INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>               [0] 7 
>>>>               INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>                 [0] 7 
>>>>               INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>                 [0] 15991 
>>>>               RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>               RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>               RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>               (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>               INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>               INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>               INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>               INFOG(6) (number of nodes in the complete tree): 3996 
>>>>               INFOG(7) (ordering option effectively use after analysis): 6 
>>>>               INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>               INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>               INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>               INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>               INFOG(12) (number of off-diagonal pivots): 19 
>>>>               INFOG(13) (number of delayed pivots after factorization): 8 
>>>>               INFOG(14) (number of memory compress after factorization): 0 
>>>>               INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>               INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>               INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>               INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>               INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>               INFOG(20) (estimated number of entries in the factors): 255801 
>>>>               INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>               INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>               INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>               INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>               INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>               INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>               INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>               INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>               INFOG(32) (after analysis: type of analysis done): 1
>>>>               INFOG(33) (value used for ICNTL(8)): 7
>>>>               INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>   linear system matrix = precond matrix:
>>>>   Mat Object:     1 MPI processes
>>>>     type: seqaij
>>>>     rows=15991, cols=15991
>>>>     total: nonzeros=223820, allocated nonzeros=431698
>>>>     total number of mallocs used during MatSetValues calls =15991
>>>>       using I-node routines: found 4000 nodes, limit used is 5
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>> 
>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> Barry,
>>>>> 
>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>> 
>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>> 
>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>>   Matt
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>> 
>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>> 
>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>> 
>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>> 
>>>>>> loop over continuation parameter
>>>>>> 
>>>>>>      SNESSolve(snes,NULL,Ucoarse);
>>>>>> 
>>>>>>      if (you decide you want to see the refined solution at this continuation point) {
>>>>>>           SNESCreate(comm,&snesrefine);
>>>>>>           SNESSetDM()
>>>>>>           etc
>>>>>>           SNESSetGridSequence(snesrefine,)
>>>>>>           SNESSolve(snesrefine,0,Ucoarse);
>>>>>>           SNESGetSolution(snesrefine,&Ufine);
>>>>>>           VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>           SNESDestroy(snesrefine);
>>>>>>     end if
>>>>>> 
>>>>>> end loop over continuation parameter.
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>> 
>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>> 
>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>> 
>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>> 
>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>> 
>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>> 
>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>> 
>>>>>>>> Call SNESSetGridSequence()
>>>>>>>> 
>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150907/8ea4ed9c/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon Sep  7 20:22:06 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Sep 2015 20:22:06 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
Message-ID: <F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>


Hmm,

   Ok you can try running it directly in the debugger since it is one process, type

  gdb ./blowup_batch_refine

  then 

  when the debugger comes up (if it does not cut and paste all output and send it)

  run -on_error_abort -snes_mf_operator  and any other options you normally use


  Barry

> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> Running with that flag gives me this:
> 
> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
> Unable to start debugger: No such file or directory
> 
> 
> 
> -gideon
> 
>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>> 
>> 
>> 
>>  Barry
>> 
>> 
>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> I?m getting an error with -snes_mf_operator, 
>>> 
>>>  0 SNES Function norm 1.421454390131e-02 
>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>> [0]PETSC ERROR: to get more information on the crash.
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Signal received
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>> 
>>> -gideon
>>> 
>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>> 
>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>> 
>>>>>     0 SNES Function norm 4.027481756921e-09 
>>>>>     1 SNES Function norm 1.760477878365e-12 
>>>>>   Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>   0 SNES Function norm 5.066222213176e+03 
>>>>>   1 SNES Function norm 8.484697184230e+02 
>>>>>   2 SNES Function norm 6.549559723294e+02 
>>>>>   3 SNES Function norm 5.770723278153e+02 
>>>>>   4 SNES Function norm 5.237702240594e+02 
>>>>>   5 SNES Function norm 4.753909019848e+02 
>>>>>   6 SNES Function norm 4.221784590755e+02 
>>>>>   7 SNES Function norm 3.806525080483e+02 
>>>>>   8 SNES Function norm 3.762054656019e+02 
>>>>>   9 SNES Function norm 3.758975226873e+02 
>>>>>  10 SNES Function norm 3.757032042706e+02 
>>>>>  11 SNES Function norm 3.728798164234e+02 
>>>>>  12 SNES Function norm 3.723078741075e+02 
>>>>>  13 SNES Function norm 3.721848059825e+02 
>>>>>  14 SNES Function norm 3.720227575629e+02 
>>>>>  15 SNES Function norm 3.720051998555e+02 
>>>>>  16 SNES Function norm 3.718945430587e+02 
>>>>>  17 SNES Function norm 3.700412694044e+02 
>>>>>  18 SNES Function norm 3.351964889461e+02 
>>>>>  19 SNES Function norm 3.096016086233e+02 
>>>>>  20 SNES Function norm 3.008410789787e+02 
>>>>>  21 SNES Function norm 2.752316716557e+02 
>>>>>  22 SNES Function norm 2.707658474165e+02 
>>>>>  23 SNES Function norm 2.698436736049e+02 
>>>>>  24 SNES Function norm 2.618233857172e+02 
>>>>>  25 SNES Function norm 2.600121920634e+02 
>>>>>  26 SNES Function norm 2.585046423168e+02 
>>>>>  27 SNES Function norm 2.568551090220e+02 
>>>>>  28 SNES Function norm 2.556404537064e+02 
>>>>>  29 SNES Function norm 2.536353523683e+02 
>>>>>  30 SNES Function norm 2.533596070171e+02 
>>>>>  31 SNES Function norm 2.532324379596e+02 
>>>>>  32 SNES Function norm 2.531842335211e+02 
>>>>>  33 SNES Function norm 2.531684527520e+02 
>>>>>  34 SNES Function norm 2.531637604618e+02 
>>>>>  35 SNES Function norm 2.531624767821e+02 
>>>>>  36 SNES Function norm 2.531621359093e+02 
>>>>>  37 SNES Function norm 2.531620504925e+02 
>>>>>  38 SNES Function norm 2.531620350055e+02 
>>>>>  39 SNES Function norm 2.531620310522e+02 
>>>>>  40 SNES Function norm 2.531620300471e+02 
>>>>>  41 SNES Function norm 2.531620298084e+02 
>>>>>  42 SNES Function norm 2.531620297478e+02 
>>>>>  43 SNES Function norm 2.531620297324e+02 
>>>>>  44 SNES Function norm 2.531620297303e+02 
>>>>>  45 SNES Function norm 2.531620297302e+02 
>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>> SNES Object: 1 MPI processes
>>>>> type: newtonls
>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>> total number of linear solver iterations=28
>>>>> total number of function evaluations=323
>>>>> total number of grid sequence refinements=2
>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>   type: bt
>>>>>     interpolation: cubic
>>>>>     alpha=1.000000e-04
>>>>>   maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>   tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>   maximum iterations=40
>>>>> KSP Object:   1 MPI processes
>>>>>   type: gmres
>>>>>     GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>     GMRES: happy breakdown tolerance 1e-30
>>>>>   maximum iterations=10000, initial guess is zero
>>>>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>   left preconditioning
>>>>>   using PRECONDITIONED norm type for convergence test
>>>>> PC Object:   1 MPI processes
>>>>>   type: lu
>>>>>     LU: out-of-place factorization
>>>>>     tolerance for zero pivot 2.22045e-14
>>>>>     matrix ordering: nd
>>>>>     factor fill ratio given 0, needed 0
>>>>>       Factored matrix follows:
>>>>>         Mat Object:           1 MPI processes
>>>>>           type: seqaij
>>>>>           rows=15991, cols=15991
>>>>>           package used to perform factorization: mumps
>>>>>           total: nonzeros=255801, allocated nonzeros=255801
>>>>>           total number of mallocs used during MatSetValues calls =0
>>>>>             MUMPS run parameters:
>>>>>               SYM (matrix type):                   0 
>>>>>               PAR (host participation):            1 
>>>>>               ICNTL(1) (output for error):         6 
>>>>>               ICNTL(2) (output of diagnostic msg): 0 
>>>>>               ICNTL(3) (output for global info):   0 
>>>>>               ICNTL(4) (level of printing):        0 
>>>>>               ICNTL(5) (input mat struct):         0 
>>>>>               ICNTL(6) (matrix prescaling):        7 
>>>>>               ICNTL(7) (sequentia matrix ordering):6 
>>>>>               ICNTL(8) (scalling strategy):        77 
>>>>>               ICNTL(10) (max num of refinements):  0 
>>>>>               ICNTL(11) (error analysis):          0 
>>>>>               ICNTL(12) (efficiency control):                         1 
>>>>>               ICNTL(13) (efficiency control):                         0 
>>>>>               ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>               ICNTL(18) (input mat struct):                           0 
>>>>>               ICNTL(19) (Shur complement info):                       0 
>>>>>               ICNTL(20) (rhs sparse pattern):                         0 
>>>>>               ICNTL(21) (somumpstion struct):                            0 
>>>>>               ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>               ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>               ICNTL(24) (detection of null pivot rows):               0 
>>>>>               ICNTL(25) (computation of a null space basis):          0 
>>>>>               ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>               ICNTL(27) (experimental parameter):                     -8 
>>>>>               ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>               ICNTL(29) (parallel ordering):                          0 
>>>>>               ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>               ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>               ICNTL(33) (compute determinant):                        0 
>>>>>               CNTL(1) (relative pivoting threshold):      0.01 
>>>>>               CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>               CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>               CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>               CNTL(5) (fixation for null pivots):         0 
>>>>>               RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>                 [0] 1.95838e+06 
>>>>>               RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>                 [0]  143924 
>>>>>               RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>                 [0]  1.95943e+06 
>>>>>               INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>               [0] 7 
>>>>>               INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>                 [0] 7 
>>>>>               INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>                 [0] 15991 
>>>>>               RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>               RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>               RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>               (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>               INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>               INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>               INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>               INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>               INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>               INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>               INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>               INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>               INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>               INFOG(12) (number of off-diagonal pivots): 19 
>>>>>               INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>               INFOG(14) (number of memory compress after factorization): 0 
>>>>>               INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>               INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>               INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>               INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>               INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>               INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>               INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>               INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>               INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>               INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>               INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>               INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>               INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>               INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>               INFOG(32) (after analysis: type of analysis done): 1
>>>>>               INFOG(33) (value used for ICNTL(8)): 7
>>>>>               INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>   linear system matrix = precond matrix:
>>>>>   Mat Object:     1 MPI processes
>>>>>     type: seqaij
>>>>>     rows=15991, cols=15991
>>>>>     total: nonzeros=223820, allocated nonzeros=431698
>>>>>     total number of mallocs used during MatSetValues calls =15991
>>>>>       using I-node routines: found 4000 nodes, limit used is 5
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>> 
>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> Barry,
>>>>>> 
>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>> 
>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>> 
>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>>   Matt
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>> 
>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>> 
>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>> 
>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>> 
>>>>>>> loop over continuation parameter
>>>>>>> 
>>>>>>>      SNESSolve(snes,NULL,Ucoarse);
>>>>>>> 
>>>>>>>      if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>           SNESCreate(comm,&snesrefine);
>>>>>>>           SNESSetDM()
>>>>>>>           etc
>>>>>>>           SNESSetGridSequence(snesrefine,)
>>>>>>>           SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>           SNESGetSolution(snesrefine,&Ufine);
>>>>>>>           VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>           SNESDestroy(snesrefine);
>>>>>>>     end if
>>>>>>> 
>>>>>>> end loop over continuation parameter.
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>> 
>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>> 
>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>> 
>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>> 
>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>> 
>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>> 
>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>> 
>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>> 
>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>> 
>>>> 
>>> 
>> 
> 


From gideon.simpson at gmail.com  Mon Sep  7 20:30:32 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 7 Sep 2015 21:30:32 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
Message-ID: <DD266ECD-90C9-46A9-ADEC-40FECF3E2393@gmail.com>

I installed the gdb-apple via macports, but now it?s throwing a fit because my application has command line flags attached to it.  Is there another way to diagnose this?

-gideon

> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
> Hmm,
> 
>   Ok you can try running it directly in the debugger since it is one process, type
> 
>  gdb ./blowup_batch_refine
> 
>  then 
> 
>  when the debugger comes up (if it does not cut and paste all output and send it)
> 
>  run -on_error_abort -snes_mf_operator  and any other options you normally use
> 
> 
>  Barry
> 
>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> Running with that flag gives me this:
>> 
>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>> Unable to start debugger: No such file or directory
>> 
>> 
>> 
>> -gideon
>> 
>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>> 
>>> 
>>> 
>>> Barry
>>> 
>>> 
>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> I?m getting an error with -snes_mf_operator, 
>>>> 
>>>> 0 SNES Function norm 1.421454390131e-02 
>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>> [0]PETSC ERROR: to get more information on the crash.
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: Signal received
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>> 
>>>>> Barry
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>> 
>>>>>>    0 SNES Function norm 4.027481756921e-09 
>>>>>>    1 SNES Function norm 1.760477878365e-12 
>>>>>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>  0 SNES Function norm 5.066222213176e+03 
>>>>>>  1 SNES Function norm 8.484697184230e+02 
>>>>>>  2 SNES Function norm 6.549559723294e+02 
>>>>>>  3 SNES Function norm 5.770723278153e+02 
>>>>>>  4 SNES Function norm 5.237702240594e+02 
>>>>>>  5 SNES Function norm 4.753909019848e+02 
>>>>>>  6 SNES Function norm 4.221784590755e+02 
>>>>>>  7 SNES Function norm 3.806525080483e+02 
>>>>>>  8 SNES Function norm 3.762054656019e+02 
>>>>>>  9 SNES Function norm 3.758975226873e+02 
>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>> SNES Object: 1 MPI processes
>>>>>> type: newtonls
>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>> total number of linear solver iterations=28
>>>>>> total number of function evaluations=323
>>>>>> total number of grid sequence refinements=2
>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>  type: bt
>>>>>>    interpolation: cubic
>>>>>>    alpha=1.000000e-04
>>>>>>  maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>  tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>  maximum iterations=40
>>>>>> KSP Object:   1 MPI processes
>>>>>>  type: gmres
>>>>>>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>    GMRES: happy breakdown tolerance 1e-30
>>>>>>  maximum iterations=10000, initial guess is zero
>>>>>>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>  left preconditioning
>>>>>>  using PRECONDITIONED norm type for convergence test
>>>>>> PC Object:   1 MPI processes
>>>>>>  type: lu
>>>>>>    LU: out-of-place factorization
>>>>>>    tolerance for zero pivot 2.22045e-14
>>>>>>    matrix ordering: nd
>>>>>>    factor fill ratio given 0, needed 0
>>>>>>      Factored matrix follows:
>>>>>>        Mat Object:           1 MPI processes
>>>>>>          type: seqaij
>>>>>>          rows=15991, cols=15991
>>>>>>          package used to perform factorization: mumps
>>>>>>          total: nonzeros=255801, allocated nonzeros=255801
>>>>>>          total number of mallocs used during MatSetValues calls =0
>>>>>>            MUMPS run parameters:
>>>>>>              SYM (matrix type):                   0 
>>>>>>              PAR (host participation):            1 
>>>>>>              ICNTL(1) (output for error):         6 
>>>>>>              ICNTL(2) (output of diagnostic msg): 0 
>>>>>>              ICNTL(3) (output for global info):   0 
>>>>>>              ICNTL(4) (level of printing):        0 
>>>>>>              ICNTL(5) (input mat struct):         0 
>>>>>>              ICNTL(6) (matrix prescaling):        7 
>>>>>>              ICNTL(7) (sequentia matrix ordering):6 
>>>>>>              ICNTL(8) (scalling strategy):        77 
>>>>>>              ICNTL(10) (max num of refinements):  0 
>>>>>>              ICNTL(11) (error analysis):          0 
>>>>>>              ICNTL(12) (efficiency control):                         1 
>>>>>>              ICNTL(13) (efficiency control):                         0 
>>>>>>              ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>              ICNTL(18) (input mat struct):                           0 
>>>>>>              ICNTL(19) (Shur complement info):                       0 
>>>>>>              ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>              ICNTL(21) (somumpstion struct):                            0 
>>>>>>              ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>              ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>              ICNTL(24) (detection of null pivot rows):               0 
>>>>>>              ICNTL(25) (computation of a null space basis):          0 
>>>>>>              ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>              ICNTL(27) (experimental parameter):                     -8 
>>>>>>              ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>              ICNTL(29) (parallel ordering):                          0 
>>>>>>              ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>              ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>              ICNTL(33) (compute determinant):                        0 
>>>>>>              CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>              CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>              CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>              CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>              CNTL(5) (fixation for null pivots):         0 
>>>>>>              RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>                [0] 1.95838e+06 
>>>>>>              RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>                [0]  143924 
>>>>>>              RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>                [0]  1.95943e+06 
>>>>>>              INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>              [0] 7 
>>>>>>              INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>                [0] 7 
>>>>>>              INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>                [0] 15991 
>>>>>>              RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>              RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>              RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>              (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>              INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>              INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>              INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>              INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>              INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>              INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>              INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>              INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>              INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>              INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>              INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>              INFOG(14) (number of memory compress after factorization): 0 
>>>>>>              INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>              INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>              INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>              INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>              INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>              INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>              INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>              INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>              INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>              INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>              INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>              INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>              INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>              INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>              INFOG(32) (after analysis: type of analysis done): 1
>>>>>>              INFOG(33) (value used for ICNTL(8)): 7
>>>>>>              INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>  linear system matrix = precond matrix:
>>>>>>  Mat Object:     1 MPI processes
>>>>>>    type: seqaij
>>>>>>    rows=15991, cols=15991
>>>>>>    total: nonzeros=223820, allocated nonzeros=431698
>>>>>>    total number of mallocs used during MatSetValues calls =15991
>>>>>>      using I-node routines: found 4000 nodes, limit used is 5
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>> 
>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> Barry,
>>>>>>> 
>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>> 
>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>> 
>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>>  Matt
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>> 
>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>> 
>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>> 
>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>> 
>>>>>>>> loop over continuation parameter
>>>>>>>> 
>>>>>>>>     SNESSolve(snes,NULL,Ucoarse);
>>>>>>>> 
>>>>>>>>     if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>          SNESCreate(comm,&snesrefine);
>>>>>>>>          SNESSetDM()
>>>>>>>>          etc
>>>>>>>>          SNESSetGridSequence(snesrefine,)
>>>>>>>>          SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>          SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>          VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>          SNESDestroy(snesrefine);
>>>>>>>>    end if
>>>>>>>> 
>>>>>>>> end loop over continuation parameter.
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>> 
>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>> 
>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>> }
>>>>>>>>>>> 
>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>> 
>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>> 
>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>> 
>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>> 
>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>> 
>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>> 
>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150907/fb5e1074/attachment-0001.html>

From gideon.simpson at gmail.com  Mon Sep  7 20:32:48 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 7 Sep 2015 21:32:48 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
Message-ID: <A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>

By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Mat type mffd
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
[0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
[0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
[0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c

-gideon

> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
> Hmm,
> 
>   Ok you can try running it directly in the debugger since it is one process, type
> 
>  gdb ./blowup_batch_refine
> 
>  then 
> 
>  when the debugger comes up (if it does not cut and paste all output and send it)
> 
>  run -on_error_abort -snes_mf_operator  and any other options you normally use
> 
> 
>  Barry
> 
>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> Running with that flag gives me this:
>> 
>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>> Unable to start debugger: No such file or directory
>> 
>> 
>> 
>> -gideon
>> 
>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>> 
>>> 
>>> 
>>> Barry
>>> 
>>> 
>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> I?m getting an error with -snes_mf_operator, 
>>>> 
>>>> 0 SNES Function norm 1.421454390131e-02 
>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>> [0]PETSC ERROR: to get more information on the crash.
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: Signal received
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>> 
>>>>> Barry
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>> 
>>>>>>    0 SNES Function norm 4.027481756921e-09 
>>>>>>    1 SNES Function norm 1.760477878365e-12 
>>>>>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>  0 SNES Function norm 5.066222213176e+03 
>>>>>>  1 SNES Function norm 8.484697184230e+02 
>>>>>>  2 SNES Function norm 6.549559723294e+02 
>>>>>>  3 SNES Function norm 5.770723278153e+02 
>>>>>>  4 SNES Function norm 5.237702240594e+02 
>>>>>>  5 SNES Function norm 4.753909019848e+02 
>>>>>>  6 SNES Function norm 4.221784590755e+02 
>>>>>>  7 SNES Function norm 3.806525080483e+02 
>>>>>>  8 SNES Function norm 3.762054656019e+02 
>>>>>>  9 SNES Function norm 3.758975226873e+02 
>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>> SNES Object: 1 MPI processes
>>>>>> type: newtonls
>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>> total number of linear solver iterations=28
>>>>>> total number of function evaluations=323
>>>>>> total number of grid sequence refinements=2
>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>  type: bt
>>>>>>    interpolation: cubic
>>>>>>    alpha=1.000000e-04
>>>>>>  maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>  tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>  maximum iterations=40
>>>>>> KSP Object:   1 MPI processes
>>>>>>  type: gmres
>>>>>>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>    GMRES: happy breakdown tolerance 1e-30
>>>>>>  maximum iterations=10000, initial guess is zero
>>>>>>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>  left preconditioning
>>>>>>  using PRECONDITIONED norm type for convergence test
>>>>>> PC Object:   1 MPI processes
>>>>>>  type: lu
>>>>>>    LU: out-of-place factorization
>>>>>>    tolerance for zero pivot 2.22045e-14
>>>>>>    matrix ordering: nd
>>>>>>    factor fill ratio given 0, needed 0
>>>>>>      Factored matrix follows:
>>>>>>        Mat Object:           1 MPI processes
>>>>>>          type: seqaij
>>>>>>          rows=15991, cols=15991
>>>>>>          package used to perform factorization: mumps
>>>>>>          total: nonzeros=255801, allocated nonzeros=255801
>>>>>>          total number of mallocs used during MatSetValues calls =0
>>>>>>            MUMPS run parameters:
>>>>>>              SYM (matrix type):                   0 
>>>>>>              PAR (host participation):            1 
>>>>>>              ICNTL(1) (output for error):         6 
>>>>>>              ICNTL(2) (output of diagnostic msg): 0 
>>>>>>              ICNTL(3) (output for global info):   0 
>>>>>>              ICNTL(4) (level of printing):        0 
>>>>>>              ICNTL(5) (input mat struct):         0 
>>>>>>              ICNTL(6) (matrix prescaling):        7 
>>>>>>              ICNTL(7) (sequentia matrix ordering):6 
>>>>>>              ICNTL(8) (scalling strategy):        77 
>>>>>>              ICNTL(10) (max num of refinements):  0 
>>>>>>              ICNTL(11) (error analysis):          0 
>>>>>>              ICNTL(12) (efficiency control):                         1 
>>>>>>              ICNTL(13) (efficiency control):                         0 
>>>>>>              ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>              ICNTL(18) (input mat struct):                           0 
>>>>>>              ICNTL(19) (Shur complement info):                       0 
>>>>>>              ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>              ICNTL(21) (somumpstion struct):                            0 
>>>>>>              ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>              ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>              ICNTL(24) (detection of null pivot rows):               0 
>>>>>>              ICNTL(25) (computation of a null space basis):          0 
>>>>>>              ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>              ICNTL(27) (experimental parameter):                     -8 
>>>>>>              ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>              ICNTL(29) (parallel ordering):                          0 
>>>>>>              ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>              ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>              ICNTL(33) (compute determinant):                        0 
>>>>>>              CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>              CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>              CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>              CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>              CNTL(5) (fixation for null pivots):         0 
>>>>>>              RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>                [0] 1.95838e+06 
>>>>>>              RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>                [0]  143924 
>>>>>>              RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>                [0]  1.95943e+06 
>>>>>>              INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>              [0] 7 
>>>>>>              INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>                [0] 7 
>>>>>>              INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>                [0] 15991 
>>>>>>              RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>              RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>              RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>              (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>              INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>              INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>              INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>              INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>              INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>              INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>              INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>              INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>              INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>              INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>              INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>              INFOG(14) (number of memory compress after factorization): 0 
>>>>>>              INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>              INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>              INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>              INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>              INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>              INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>              INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>              INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>              INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>              INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>              INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>              INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>              INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>              INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>              INFOG(32) (after analysis: type of analysis done): 1
>>>>>>              INFOG(33) (value used for ICNTL(8)): 7
>>>>>>              INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>  linear system matrix = precond matrix:
>>>>>>  Mat Object:     1 MPI processes
>>>>>>    type: seqaij
>>>>>>    rows=15991, cols=15991
>>>>>>    total: nonzeros=223820, allocated nonzeros=431698
>>>>>>    total number of mallocs used during MatSetValues calls =15991
>>>>>>      using I-node routines: found 4000 nodes, limit used is 5
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>> 
>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> Barry,
>>>>>>> 
>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>> 
>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>> 
>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>>  Matt
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>> 
>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>> 
>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>> 
>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>> 
>>>>>>>> loop over continuation parameter
>>>>>>>> 
>>>>>>>>     SNESSolve(snes,NULL,Ucoarse);
>>>>>>>> 
>>>>>>>>     if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>          SNESCreate(comm,&snesrefine);
>>>>>>>>          SNESSetDM()
>>>>>>>>          etc
>>>>>>>>          SNESSetGridSequence(snesrefine,)
>>>>>>>>          SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>          SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>          VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>          SNESDestroy(snesrefine);
>>>>>>>>    end if
>>>>>>>> 
>>>>>>>> end loop over continuation parameter.
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>> 
>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>> 
>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>> }
>>>>>>>>>>> 
>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>> 
>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>> 
>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>> 
>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>> 
>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>> 
>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>> 
>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150907/a5d515d9/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon Sep  7 20:35:46 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Sep 2015 20:35:46 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <DD266ECD-90C9-46A9-ADEC-40FECF3E2393@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<DD266ECD-90C9-46A9-ADEC-40FECF3E2393@gmail.com>
Message-ID: <2409A4BC-2B27-4F75-ABE8-4E375BE6DEA0@mcs.anl.gov>


  Oh sorry, if you are using the standard Apple software stack you should use lldb instead of gdb

   Barry

> On Sep 7, 2015, at 8:30 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I installed the gdb-apple via macports, but now it?s throwing a fit because my application has command line flags attached to it.  Is there another way to diagnose this?
> 
> -gideon
> 
>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>> Hmm,
>> 
>>   Ok you can try running it directly in the debugger since it is one process, type
>> 
>>  gdb ./blowup_batch_refine
>> 
>>  then 
>> 
>>  when the debugger comes up (if it does not cut and paste all output and send it)
>> 
>>  run -on_error_abort -snes_mf_operator  and any other options you normally use
>> 
>> 
>>  Barry
>> 
>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> Running with that flag gives me this:
>>> 
>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>> Unable to start debugger: No such file or directory
>>> 
>>> 
>>> 
>>> -gideon
>>> 
>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>> 
>>>> 
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> I?m getting an error with -snes_mf_operator, 
>>>>> 
>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>> [0]PETSC ERROR: Signal received
>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>> 
>>>>>>>    0 SNES Function norm 4.027481756921e-09 
>>>>>>>    1 SNES Function norm 1.760477878365e-12 
>>>>>>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>  0 SNES Function norm 5.066222213176e+03 
>>>>>>>  1 SNES Function norm 8.484697184230e+02 
>>>>>>>  2 SNES Function norm 6.549559723294e+02 
>>>>>>>  3 SNES Function norm 5.770723278153e+02 
>>>>>>>  4 SNES Function norm 5.237702240594e+02 
>>>>>>>  5 SNES Function norm 4.753909019848e+02 
>>>>>>>  6 SNES Function norm 4.221784590755e+02 
>>>>>>>  7 SNES Function norm 3.806525080483e+02 
>>>>>>>  8 SNES Function norm 3.762054656019e+02 
>>>>>>>  9 SNES Function norm 3.758975226873e+02 
>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>> SNES Object: 1 MPI processes
>>>>>>> type: newtonls
>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>> total number of linear solver iterations=28
>>>>>>> total number of function evaluations=323
>>>>>>> total number of grid sequence refinements=2
>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>  type: bt
>>>>>>>    interpolation: cubic
>>>>>>>    alpha=1.000000e-04
>>>>>>>  maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>  tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>  maximum iterations=40
>>>>>>> KSP Object:   1 MPI processes
>>>>>>>  type: gmres
>>>>>>>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>    GMRES: happy breakdown tolerance 1e-30
>>>>>>>  maximum iterations=10000, initial guess is zero
>>>>>>>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>  left preconditioning
>>>>>>>  using PRECONDITIONED norm type for convergence test
>>>>>>> PC Object:   1 MPI processes
>>>>>>>  type: lu
>>>>>>>    LU: out-of-place factorization
>>>>>>>    tolerance for zero pivot 2.22045e-14
>>>>>>>    matrix ordering: nd
>>>>>>>    factor fill ratio given 0, needed 0
>>>>>>>      Factored matrix follows:
>>>>>>>        Mat Object:           1 MPI processes
>>>>>>>          type: seqaij
>>>>>>>          rows=15991, cols=15991
>>>>>>>          package used to perform factorization: mumps
>>>>>>>          total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>          total number of mallocs used during MatSetValues calls =0
>>>>>>>            MUMPS run parameters:
>>>>>>>              SYM (matrix type):                   0 
>>>>>>>              PAR (host participation):            1 
>>>>>>>              ICNTL(1) (output for error):         6 
>>>>>>>              ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>              ICNTL(3) (output for global info):   0 
>>>>>>>              ICNTL(4) (level of printing):        0 
>>>>>>>              ICNTL(5) (input mat struct):         0 
>>>>>>>              ICNTL(6) (matrix prescaling):        7 
>>>>>>>              ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>              ICNTL(8) (scalling strategy):        77 
>>>>>>>              ICNTL(10) (max num of refinements):  0 
>>>>>>>              ICNTL(11) (error analysis):          0 
>>>>>>>              ICNTL(12) (efficiency control):                         1 
>>>>>>>              ICNTL(13) (efficiency control):                         0 
>>>>>>>              ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>              ICNTL(18) (input mat struct):                           0 
>>>>>>>              ICNTL(19) (Shur complement info):                       0 
>>>>>>>              ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>              ICNTL(21) (somumpstion struct):                            0 
>>>>>>>              ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>              ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>              ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>              ICNTL(25) (computation of a null space basis):          0 
>>>>>>>              ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>              ICNTL(27) (experimental parameter):                     -8 
>>>>>>>              ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>              ICNTL(29) (parallel ordering):                          0 
>>>>>>>              ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>              ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>              ICNTL(33) (compute determinant):                        0 
>>>>>>>              CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>              CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>              CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>              CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>              CNTL(5) (fixation for null pivots):         0 
>>>>>>>              RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>                [0] 1.95838e+06 
>>>>>>>              RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>                [0]  143924 
>>>>>>>              RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>                [0]  1.95943e+06 
>>>>>>>              INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>              [0] 7 
>>>>>>>              INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>                [0] 7 
>>>>>>>              INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>                [0] 15991 
>>>>>>>              RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>              RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>              RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>              (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>              INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>              INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>              INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>              INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>              INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>              INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>              INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>              INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>              INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>              INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>              INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>              INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>              INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>              INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>              INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>              INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>              INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>              INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>              INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>              INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>              INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>              INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>              INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>              INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>              INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>              INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>              INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>              INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>              INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>  linear system matrix = precond matrix:
>>>>>>>  Mat Object:     1 MPI processes
>>>>>>>    type: seqaij
>>>>>>>    rows=15991, cols=15991
>>>>>>>    total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>    total number of mallocs used during MatSetValues calls =15991
>>>>>>>      using I-node routines: found 4000 nodes, limit used is 5
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> Barry,
>>>>>>>> 
>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>> 
>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>> 
>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>>  Matt
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>> 
>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>> 
>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>> 
>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>> 
>>>>>>>>> loop over continuation parameter
>>>>>>>>> 
>>>>>>>>>     SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>> 
>>>>>>>>>     if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>          SNESCreate(comm,&snesrefine);
>>>>>>>>>          SNESSetDM()
>>>>>>>>>          etc
>>>>>>>>>          SNESSetGridSequence(snesrefine,)
>>>>>>>>>          SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>          SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>          VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>          SNESDestroy(snesrefine);
>>>>>>>>>    end if
>>>>>>>>> 
>>>>>>>>> end loop over continuation parameter.
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>> 
>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>> 
>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>> }
>>>>>>>>>>>> 
>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>> 
>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>> 
>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>> 
>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>> 
>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>> 
>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>> 
>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From bsmith at mcs.anl.gov  Mon Sep  7 20:39:28 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Sep 2015 20:39:28 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
Message-ID: <81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>


  This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have

   ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)

  you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c

  This is a minor glitch we'll get past.

  Barry

> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Mat type mffd
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
> 
> -gideon
> 
>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>> Hmm,
>> 
>>   Ok you can try running it directly in the debugger since it is one process, type
>> 
>>  gdb ./blowup_batch_refine
>> 
>>  then 
>> 
>>  when the debugger comes up (if it does not cut and paste all output and send it)
>> 
>>  run -on_error_abort -snes_mf_operator  and any other options you normally use
>> 
>> 
>>  Barry
>> 
>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> Running with that flag gives me this:
>>> 
>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>> Unable to start debugger: No such file or directory
>>> 
>>> 
>>> 
>>> -gideon
>>> 
>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>> 
>>>> 
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> I?m getting an error with -snes_mf_operator, 
>>>>> 
>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>> [0]PETSC ERROR: Signal received
>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>> 
>>>>>>>    0 SNES Function norm 4.027481756921e-09 
>>>>>>>    1 SNES Function norm 1.760477878365e-12 
>>>>>>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>  0 SNES Function norm 5.066222213176e+03 
>>>>>>>  1 SNES Function norm 8.484697184230e+02 
>>>>>>>  2 SNES Function norm 6.549559723294e+02 
>>>>>>>  3 SNES Function norm 5.770723278153e+02 
>>>>>>>  4 SNES Function norm 5.237702240594e+02 
>>>>>>>  5 SNES Function norm 4.753909019848e+02 
>>>>>>>  6 SNES Function norm 4.221784590755e+02 
>>>>>>>  7 SNES Function norm 3.806525080483e+02 
>>>>>>>  8 SNES Function norm 3.762054656019e+02 
>>>>>>>  9 SNES Function norm 3.758975226873e+02 
>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>> SNES Object: 1 MPI processes
>>>>>>> type: newtonls
>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>> total number of linear solver iterations=28
>>>>>>> total number of function evaluations=323
>>>>>>> total number of grid sequence refinements=2
>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>  type: bt
>>>>>>>    interpolation: cubic
>>>>>>>    alpha=1.000000e-04
>>>>>>>  maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>  tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>  maximum iterations=40
>>>>>>> KSP Object:   1 MPI processes
>>>>>>>  type: gmres
>>>>>>>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>    GMRES: happy breakdown tolerance 1e-30
>>>>>>>  maximum iterations=10000, initial guess is zero
>>>>>>>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>  left preconditioning
>>>>>>>  using PRECONDITIONED norm type for convergence test
>>>>>>> PC Object:   1 MPI processes
>>>>>>>  type: lu
>>>>>>>    LU: out-of-place factorization
>>>>>>>    tolerance for zero pivot 2.22045e-14
>>>>>>>    matrix ordering: nd
>>>>>>>    factor fill ratio given 0, needed 0
>>>>>>>      Factored matrix follows:
>>>>>>>        Mat Object:           1 MPI processes
>>>>>>>          type: seqaij
>>>>>>>          rows=15991, cols=15991
>>>>>>>          package used to perform factorization: mumps
>>>>>>>          total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>          total number of mallocs used during MatSetValues calls =0
>>>>>>>            MUMPS run parameters:
>>>>>>>              SYM (matrix type):                   0 
>>>>>>>              PAR (host participation):            1 
>>>>>>>              ICNTL(1) (output for error):         6 
>>>>>>>              ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>              ICNTL(3) (output for global info):   0 
>>>>>>>              ICNTL(4) (level of printing):        0 
>>>>>>>              ICNTL(5) (input mat struct):         0 
>>>>>>>              ICNTL(6) (matrix prescaling):        7 
>>>>>>>              ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>              ICNTL(8) (scalling strategy):        77 
>>>>>>>              ICNTL(10) (max num of refinements):  0 
>>>>>>>              ICNTL(11) (error analysis):          0 
>>>>>>>              ICNTL(12) (efficiency control):                         1 
>>>>>>>              ICNTL(13) (efficiency control):                         0 
>>>>>>>              ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>              ICNTL(18) (input mat struct):                           0 
>>>>>>>              ICNTL(19) (Shur complement info):                       0 
>>>>>>>              ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>              ICNTL(21) (somumpstion struct):                            0 
>>>>>>>              ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>              ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>              ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>              ICNTL(25) (computation of a null space basis):          0 
>>>>>>>              ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>              ICNTL(27) (experimental parameter):                     -8 
>>>>>>>              ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>              ICNTL(29) (parallel ordering):                          0 
>>>>>>>              ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>              ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>              ICNTL(33) (compute determinant):                        0 
>>>>>>>              CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>              CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>              CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>              CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>              CNTL(5) (fixation for null pivots):         0 
>>>>>>>              RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>                [0] 1.95838e+06 
>>>>>>>              RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>                [0]  143924 
>>>>>>>              RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>                [0]  1.95943e+06 
>>>>>>>              INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>              [0] 7 
>>>>>>>              INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>                [0] 7 
>>>>>>>              INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>                [0] 15991 
>>>>>>>              RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>              RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>              RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>              (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>              INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>              INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>              INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>              INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>              INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>              INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>              INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>              INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>              INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>              INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>              INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>              INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>              INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>              INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>              INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>              INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>              INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>              INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>              INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>              INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>              INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>              INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>              INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>              INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>              INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>              INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>              INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>              INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>              INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>  linear system matrix = precond matrix:
>>>>>>>  Mat Object:     1 MPI processes
>>>>>>>    type: seqaij
>>>>>>>    rows=15991, cols=15991
>>>>>>>    total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>    total number of mallocs used during MatSetValues calls =15991
>>>>>>>      using I-node routines: found 4000 nodes, limit used is 5
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> Barry,
>>>>>>>> 
>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>> 
>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>> 
>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>>  Matt
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>> 
>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>> 
>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>> 
>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>> 
>>>>>>>>> loop over continuation parameter
>>>>>>>>> 
>>>>>>>>>     SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>> 
>>>>>>>>>     if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>          SNESCreate(comm,&snesrefine);
>>>>>>>>>          SNESSetDM()
>>>>>>>>>          etc
>>>>>>>>>          SNESSetGridSequence(snesrefine,)
>>>>>>>>>          SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>          SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>          VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>          SNESDestroy(snesrefine);
>>>>>>>>>    end if
>>>>>>>>> 
>>>>>>>>> end loop over continuation parameter.
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>> 
>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>> 
>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>> }
>>>>>>>>>>>> 
>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>> 
>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>> 
>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>> 
>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>> 
>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>> 
>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>> 
>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From gideon.simpson at gmail.com  Mon Sep  7 20:49:07 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 7 Sep 2015 21:49:07 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
Message-ID: <CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>

Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:

      0 SNES Function norm 1.132185384796e-08 
        0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
        1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
      1 SNES Function norm 2.177599365111e-12 
    Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
    0 SNES Function norm 5.066222213176e+03 
      0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
    1 SNES Function norm 8.482593852817e+02 
      0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
      2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
    2 SNES Function norm 6.543140468549e+02 
      0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
      2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
    3 SNES Function norm 5.766430557220e+02 
      0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
      2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
    4 SNES Function norm 5.235211958260e+02 
      0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
      2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
    5 SNES Function norm 4.752913229649e+02 
      0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
      2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
    6 SNES Function norm 4.220255380391e+02 
      0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
      2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
    7 SNES Function norm 3.805408907074e+02 
      0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
    8 SNES Function norm 3.764619752339e+02 
      0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
      2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
    9 SNES Function norm 3.761182227091e+02 
      0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
      2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
   10 SNES Function norm 3.740017190063e+02 
      0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
      2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
   11 SNES Function norm 3.725903477238e+02 
      0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
      2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
   12 SNES Function norm 3.716162097231e+02 
      0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
      2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
   13 SNES Function norm 3.674168632847e+02 
      0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
      2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
   14 SNES Function norm 3.532395445266e+02 
      0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
      2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
   15 SNES Function norm 3.182438872366e+02 
      0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
      2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
   16 SNES Function norm 3.091759892779e+02 
      0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
      2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
   17 SNES Function norm 2.987839504359e+02 
      0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
      2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
   18 SNES Function norm 2.987073622777e+02 
      0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
      2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
   19 SNES Function norm 2.987067936734e+02 
      0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
      2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
   20 SNES Function norm 2.987067502910e+02 
      0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
      2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
   21 SNES Function norm 2.987064584431e+02 
      0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
      2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
   22 SNES Function norm 2.987064525262e+02 
      0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
      2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
   23 SNES Function norm 2.987064121622e+02 
      0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
      2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
   24 SNES Function norm 2.987063973426e+02 
      0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
      2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
   25 SNES Function norm 2.987063920553e+02 
      0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
      2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
   26 SNES Function norm 2.987063919786e+02 
      0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
      2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
   27 SNES Function norm 2.987063393145e+02 
      0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
      2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
   28 SNES Function norm 2.987063253277e+02 
      0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
      2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
   29 SNES Function norm 2.987063197153e+02 
      0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
      2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
   30 SNES Function norm 2.987063193089e+02 
      0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
      2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
   31 SNES Function norm 2.987063192061e+02 
      0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
      2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
   32 SNES Function norm 2.987063094492e+02 
      0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
      2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
   33 SNES Function norm 2.987063089942e+02 
      0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
      2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
   34 SNES Function norm 2.987062660109e+02 
      0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
      2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
   35 SNES Function norm 2.987062107900e+02 
      0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
      2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
   36 SNES Function norm 2.987062055224e+02 
      0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
      2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
   37 SNES Function norm 2.987061774798e+02 
      0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
      2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
   38 SNES Function norm 2.987061715400e+02 
      0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
      2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
   39 SNES Function norm 2.987061699634e+02 
      0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
      2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
   40 SNES Function norm 2.987061630064e+02 
      0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
   41 SNES Function norm 2.987061630064e+02 
      0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
   42 SNES Function norm 2.987061630064e+02 
      0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
   43 SNES Function norm 2.987061630064e+02 
      0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
   44 SNES Function norm 2.987061630064e+02 
      0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
   45 SNES Function norm 2.987061630064e+02 
      0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
   46 SNES Function norm 2.987061630064e+02 
      0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
   47 SNES Function norm 2.987061630064e+02 
      0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
  Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47


-gideon

> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
> 
>   ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
> 
>  you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
> 
>  This is a minor glitch we'll get past.
> 
>  Barry
> 
>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>> 
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: No support for this operation for this object type
>> [0]PETSC ERROR: Mat type mffd
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>> 
>> -gideon
>> 
>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> Hmm,
>>> 
>>>  Ok you can try running it directly in the debugger since it is one process, type
>>> 
>>> gdb ./blowup_batch_refine
>>> 
>>> then 
>>> 
>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>> 
>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>> 
>>> 
>>> Barry
>>> 
>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> Running with that flag gives me this:
>>>> 
>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>> Unable to start debugger: No such file or directory
>>>> 
>>>> 
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>> 
>>>>> 
>>>>> 
>>>>> Barry
>>>>> 
>>>>> 
>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>> 
>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>> [0]PETSC ERROR: Signal received
>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>> 
>>>>>>>>   0 SNES Function norm 4.027481756921e-09 
>>>>>>>>   1 SNES Function norm 1.760477878365e-12 
>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>> type: newtonls
>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>> total number of linear solver iterations=28
>>>>>>>> total number of function evaluations=323
>>>>>>>> total number of grid sequence refinements=2
>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>> type: bt
>>>>>>>>   interpolation: cubic
>>>>>>>>   alpha=1.000000e-04
>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>> maximum iterations=40
>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>> type: gmres
>>>>>>>>   GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>   GMRES: happy breakdown tolerance 1e-30
>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>> left preconditioning
>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>> PC Object:   1 MPI processes
>>>>>>>> type: lu
>>>>>>>>   LU: out-of-place factorization
>>>>>>>>   tolerance for zero pivot 2.22045e-14
>>>>>>>>   matrix ordering: nd
>>>>>>>>   factor fill ratio given 0, needed 0
>>>>>>>>     Factored matrix follows:
>>>>>>>>       Mat Object:           1 MPI processes
>>>>>>>>         type: seqaij
>>>>>>>>         rows=15991, cols=15991
>>>>>>>>         package used to perform factorization: mumps
>>>>>>>>         total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>         total number of mallocs used during MatSetValues calls =0
>>>>>>>>           MUMPS run parameters:
>>>>>>>>             SYM (matrix type):                   0 
>>>>>>>>             PAR (host participation):            1 
>>>>>>>>             ICNTL(1) (output for error):         6 
>>>>>>>>             ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>             ICNTL(3) (output for global info):   0 
>>>>>>>>             ICNTL(4) (level of printing):        0 
>>>>>>>>             ICNTL(5) (input mat struct):         0 
>>>>>>>>             ICNTL(6) (matrix prescaling):        7 
>>>>>>>>             ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>             ICNTL(8) (scalling strategy):        77 
>>>>>>>>             ICNTL(10) (max num of refinements):  0 
>>>>>>>>             ICNTL(11) (error analysis):          0 
>>>>>>>>             ICNTL(12) (efficiency control):                         1 
>>>>>>>>             ICNTL(13) (efficiency control):                         0 
>>>>>>>>             ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>             ICNTL(18) (input mat struct):                           0 
>>>>>>>>             ICNTL(19) (Shur complement info):                       0 
>>>>>>>>             ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>             ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>             ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>             ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>             ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>             ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>             ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>             ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>             ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>             ICNTL(29) (parallel ordering):                          0 
>>>>>>>>             ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>             ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>             ICNTL(33) (compute determinant):                        0 
>>>>>>>>             CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>             CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>             CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>             CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>             CNTL(5) (fixation for null pivots):         0 
>>>>>>>>             RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>               [0] 1.95838e+06 
>>>>>>>>             RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>               [0]  143924 
>>>>>>>>             RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>               [0]  1.95943e+06 
>>>>>>>>             INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>             [0] 7 
>>>>>>>>             INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>               [0] 7 
>>>>>>>>             INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>               [0] 15991 
>>>>>>>>             RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>             RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>             RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>             (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>             INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>             INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>             INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>             INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>             INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>             INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>             INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>             INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>             INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>             INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>             INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>             INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>             INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>             INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>             INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>             INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>             INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>             INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>             INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>             INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>             INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>             INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>             INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>             INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>             INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>             INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>             INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>             INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>             INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>> linear system matrix = precond matrix:
>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>   type: seqaij
>>>>>>>>   rows=15991, cols=15991
>>>>>>>>   total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>   total number of mallocs used during MatSetValues calls =15991
>>>>>>>>     using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> Barry,
>>>>>>>>> 
>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>> 
>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>> 
>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> Matt
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>> 
>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>> 
>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>> 
>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>> 
>>>>>>>>>> loop over continuation parameter
>>>>>>>>>> 
>>>>>>>>>>    SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>> 
>>>>>>>>>>    if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>         SNESCreate(comm,&snesrefine);
>>>>>>>>>>         SNESSetDM()
>>>>>>>>>>         etc
>>>>>>>>>>         SNESSetGridSequence(snesrefine,)
>>>>>>>>>>         SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>         SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>         VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>         SNESDestroy(snesrefine);
>>>>>>>>>>   end if
>>>>>>>>>> 
>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>> }
>>>>>>>>>>>>> 
>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>> 
>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>> 
>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>> 
>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>> 
>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>> 
>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>> 
>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150907/62f0728c/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon Sep  7 21:03:53 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Sep 2015 21:03:53 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
Message-ID: <96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>


  Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)

  If not run with with -ksp_type fgmres and send the same output.

  Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.

  Barry


   21 SNES Function norm 2.987064584431e+02 
      0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
      2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01


> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
> 
>       0 SNES Function norm 1.132185384796e-08 
>         0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>         1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>       1 SNES Function norm 2.177599365111e-12 
>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>     0 SNES Function norm 5.066222213176e+03 
>       0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>     1 SNES Function norm 8.482593852817e+02 
>       0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>       2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>     2 SNES Function norm 6.543140468549e+02 
>       0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>       2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>     3 SNES Function norm 5.766430557220e+02 
>       0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>       2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>     4 SNES Function norm 5.235211958260e+02 
>       0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>       2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>     5 SNES Function norm 4.752913229649e+02 
>       0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>       2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>     6 SNES Function norm 4.220255380391e+02 
>       0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>       2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>     7 SNES Function norm 3.805408907074e+02 
>       0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>     8 SNES Function norm 3.764619752339e+02 
>       0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>       2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>     9 SNES Function norm 3.761182227091e+02 
>       0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>       2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>    10 SNES Function norm 3.740017190063e+02 
>       0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>       2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>    11 SNES Function norm 3.725903477238e+02 
>       0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>       2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>    12 SNES Function norm 3.716162097231e+02 
>       0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>       2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>    13 SNES Function norm 3.674168632847e+02 
>       0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>       2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>    14 SNES Function norm 3.532395445266e+02 
>       0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>       2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>    15 SNES Function norm 3.182438872366e+02 
>       0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>       2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>    16 SNES Function norm 3.091759892779e+02 
>       0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>       2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>    17 SNES Function norm 2.987839504359e+02 
>       0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>       2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>    18 SNES Function norm 2.987073622777e+02 
>       0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>       2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>    19 SNES Function norm 2.987067936734e+02 
>       0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>       2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>    20 SNES Function norm 2.987067502910e+02 
>       0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>       2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>    21 SNES Function norm 2.987064584431e+02 
>       0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>       2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>    22 SNES Function norm 2.987064525262e+02 
>       0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>       2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>    23 SNES Function norm 2.987064121622e+02 
>       0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>       2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>    24 SNES Function norm 2.987063973426e+02 
>       0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>       2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>    25 SNES Function norm 2.987063920553e+02 
>       0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>       2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>    26 SNES Function norm 2.987063919786e+02 
>       0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>       2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>    27 SNES Function norm 2.987063393145e+02 
>       0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>       2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>    28 SNES Function norm 2.987063253277e+02 
>       0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>       2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>    29 SNES Function norm 2.987063197153e+02 
>       0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>       2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>    30 SNES Function norm 2.987063193089e+02 
>       0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>       2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>    31 SNES Function norm 2.987063192061e+02 
>       0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>       2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>    32 SNES Function norm 2.987063094492e+02 
>       0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>       2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>    33 SNES Function norm 2.987063089942e+02 
>       0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>       2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>    34 SNES Function norm 2.987062660109e+02 
>       0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>       2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>    35 SNES Function norm 2.987062107900e+02 
>       0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>       2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>    36 SNES Function norm 2.987062055224e+02 
>       0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>       2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>    37 SNES Function norm 2.987061774798e+02 
>       0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>       2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>    38 SNES Function norm 2.987061715400e+02 
>       0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>       2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>    39 SNES Function norm 2.987061699634e+02 
>       0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>       2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>    40 SNES Function norm 2.987061630064e+02 
>       0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>    41 SNES Function norm 2.987061630064e+02 
>       0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>    42 SNES Function norm 2.987061630064e+02 
>       0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>    43 SNES Function norm 2.987061630064e+02 
>       0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>    44 SNES Function norm 2.987061630064e+02 
>       0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>    45 SNES Function norm 2.987061630064e+02 
>       0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>    46 SNES Function norm 2.987061630064e+02 
>       0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>    47 SNES Function norm 2.987061630064e+02 
>       0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
> 
> 
> -gideon
> 
>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>> 
>>   ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>> 
>>  you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>> 
>>  This is a minor glitch we'll get past.
>> 
>>  Barry
>> 
>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>> 
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: No support for this operation for this object type
>>> [0]PETSC ERROR: Mat type mffd
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>> 
>>> -gideon
>>> 
>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> Hmm,
>>>> 
>>>>  Ok you can try running it directly in the debugger since it is one process, type
>>>> 
>>>> gdb ./blowup_batch_refine
>>>> 
>>>> then 
>>>> 
>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>> 
>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>> 
>>>> 
>>>> Barry
>>>> 
>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> Running with that flag gives me this:
>>>>> 
>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>> Unable to start debugger: No such file or directory
>>>>> 
>>>>> 
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> 
>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>> 
>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>> 
>>>>>>>>>   0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>   1 SNES Function norm 1.760477878365e-12 
>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>> type: newtonls
>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>> total number of function evaluations=323
>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>> type: bt
>>>>>>>>>   interpolation: cubic
>>>>>>>>>   alpha=1.000000e-04
>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>> maximum iterations=40
>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>> type: gmres
>>>>>>>>>   GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>   GMRES: happy breakdown tolerance 1e-30
>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>> left preconditioning
>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>> type: lu
>>>>>>>>>   LU: out-of-place factorization
>>>>>>>>>   tolerance for zero pivot 2.22045e-14
>>>>>>>>>   matrix ordering: nd
>>>>>>>>>   factor fill ratio given 0, needed 0
>>>>>>>>>     Factored matrix follows:
>>>>>>>>>       Mat Object:           1 MPI processes
>>>>>>>>>         type: seqaij
>>>>>>>>>         rows=15991, cols=15991
>>>>>>>>>         package used to perform factorization: mumps
>>>>>>>>>         total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>         total number of mallocs used during MatSetValues calls =0
>>>>>>>>>           MUMPS run parameters:
>>>>>>>>>             SYM (matrix type):                   0 
>>>>>>>>>             PAR (host participation):            1 
>>>>>>>>>             ICNTL(1) (output for error):         6 
>>>>>>>>>             ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>             ICNTL(3) (output for global info):   0 
>>>>>>>>>             ICNTL(4) (level of printing):        0 
>>>>>>>>>             ICNTL(5) (input mat struct):         0 
>>>>>>>>>             ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>             ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>             ICNTL(8) (scalling strategy):        77 
>>>>>>>>>             ICNTL(10) (max num of refinements):  0 
>>>>>>>>>             ICNTL(11) (error analysis):          0 
>>>>>>>>>             ICNTL(12) (efficiency control):                         1 
>>>>>>>>>             ICNTL(13) (efficiency control):                         0 
>>>>>>>>>             ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>             ICNTL(18) (input mat struct):                           0 
>>>>>>>>>             ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>             ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>             ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>             ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>             ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>             ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>             ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>             ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>             ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>             ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>             ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>             ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>             ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>             ICNTL(33) (compute determinant):                        0 
>>>>>>>>>             CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>             CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>             CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>             CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>             CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>             RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>               [0] 1.95838e+06 
>>>>>>>>>             RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>               [0]  143924 
>>>>>>>>>             RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>               [0]  1.95943e+06 
>>>>>>>>>             INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>             [0] 7 
>>>>>>>>>             INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>               [0] 7 
>>>>>>>>>             INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>               [0] 15991 
>>>>>>>>>             RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>             RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>             RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>             (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>             INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>             INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>             INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>             INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>             INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>             INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>             INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>             INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>             INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>             INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>             INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>             INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>             INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>             INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>             INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>             INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>             INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>             INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>             INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>             INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>             INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>             INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>             INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>             INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>             INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>             INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>             INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>             INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>             INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>   type: seqaij
>>>>>>>>>   rows=15991, cols=15991
>>>>>>>>>   total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>   total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>     using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> Barry,
>>>>>>>>>> 
>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>> 
>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>> 
>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> 
>>>>>>>>>> Matt
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>> 
>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>> 
>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>> 
>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>> 
>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>> 
>>>>>>>>>>>    SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>> 
>>>>>>>>>>>    if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>         SNESCreate(comm,&snesrefine);
>>>>>>>>>>>         SNESSetDM()
>>>>>>>>>>>         etc
>>>>>>>>>>>         SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>         SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>         SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>         VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>         SNESDestroy(snesrefine);
>>>>>>>>>>>   end if
>>>>>>>>>>> 
>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -- 
>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From gideon.simpson at gmail.com  Mon Sep  7 21:16:57 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 7 Sep 2015 22:16:57 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
Message-ID: <5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>

Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.

When I run with -ksp_type fgmres, I get

      0 SNES Function norm 3.857327250267e-09 
        0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
        1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
        2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
      1 SNES Function norm 2.479697868412e-12 
    Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
    0 SNES Function norm 5.066222213176e+03 
      0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
    1 SNES Function norm 8.484696862318e+02 
      0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
      2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
    2 SNES Function norm 6.551139809777e+02 
      0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
      2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
    3 SNES Function norm 5.771483337231e+02 
      0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
      2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
    4 SNES Function norm 5.237540245670e+02 
      0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
      2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
    5 SNES Function norm 4.753058321803e+02 
      0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
      2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
    6 SNES Function norm 4.221482124411e+02 
      0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
      2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
    7 SNES Function norm 3.803620028964e+02 
      0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
      2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
    8 SNES Function norm 3.769519577570e+02 
      0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
      2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
      3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
    9 SNES Function norm 3.766389203406e+02 
      0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
      2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
   10 SNES Function norm 3.751822489648e+02 
      0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
      2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
   11 SNES Function norm 3.737189963925e+02 
      0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
      2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
      3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
   12 SNES Function norm 3.737110331594e+02 
      0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
      2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
   13 SNES Function norm 3.684175513154e+02 
      0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
      2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
   14 SNES Function norm 3.509930783933e+02 
      0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
      2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
   15 SNES Function norm 3.162121096565e+02 
      0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
      2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
   16 SNES Function norm 2.890165052818e+02 
      0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
      2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
   17 SNES Function norm 2.816929421445e+02 
      0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
      2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
      3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
   18 SNES Function norm 2.814407559738e+02 
      0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
      2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
   19 SNES Function norm 2.810261821835e+02 
      0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
      2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
   20 SNES Function norm 2.755723672077e+02 
      0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
      2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
      3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
      4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
      5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
      6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
   21 SNES Function norm 2.755720389981e+02 
      0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
      2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
      3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
      4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
      5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
      6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
   22 SNES Function norm 2.755716464881e+02 
      0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
      2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
      3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
      4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
   23 SNES Function norm 2.755713163168e+02 
      0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
      2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
      3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
      4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
      5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
      6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
      7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
      8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
   24 SNES Function norm 2.755712363532e+02 
      0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
      2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
      3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
      4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
      5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
      6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
      7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
      8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
      9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
     10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
     11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
     12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
     13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
     14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
     15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
     16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
     17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
     18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
     19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
     20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
     21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
     22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
     23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
     24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
     25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
     26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
     27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
     28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
     29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
     30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
     31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
     32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
     33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
     34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
     35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
     36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
     37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
     38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
     39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
     40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
     41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
     42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
     43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
     44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
     45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
     46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
     47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
     48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
     49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
     50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
     51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
     52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
     53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
     54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
     55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
     56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
     57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
     58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
     59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
     60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
     61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
     62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
     63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
     64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
     65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
     66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
     67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
     68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
     69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
     70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
     71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
     72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
     73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
     74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
     75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
     76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
     77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
     78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
     79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
     80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
     81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
     82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
     83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
     84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
     85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
     86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
     87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
     88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
     89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
     90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
     91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
     92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
     93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
     94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
     95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
     96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
     97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
     98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
     99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
    100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
    101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
    102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
    103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
    104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
    105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
    106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
    107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
    108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
    109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
    110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
    111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
    112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
    113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
    114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
    115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
    116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
    117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
    118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
    119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
    120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
    121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
    122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
    123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
    124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
    125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
    126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
    127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
    128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
    129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
    130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
    131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
    132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
    133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
    134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
    135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
    136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
    137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
    138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
    139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
    140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
    141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
    142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
    143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
    144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
    145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
    146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
    147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
    148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
    149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
    150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
    151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
    152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
    153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
    154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
    155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
    156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
    157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
    158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
    159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
    160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
    161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
    162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
    163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
    164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
    165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
    166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
    167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
    168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
    169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
    170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
    171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
    172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
    173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
    174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
    175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
    176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
    177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
    178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
    179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
    180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
    181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
    182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
    183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
    184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
    185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
    186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
    187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
    188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
    189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
    190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
    191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
    192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
    193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
    194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
    195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
    196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
    197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
    198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
    199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
    200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
    201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
    202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
    203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
    204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
    205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
    206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
    207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
    208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
    209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
    210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
  Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24

-gideon

> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
> 
>  If not run with with -ksp_type fgmres and send the same output.
> 
>  Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
> 
>  Barry
> 
> 
>   21 SNES Function norm 2.987064584431e+02 
>      0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>      1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>      2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
> 
> 
>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>> 
>>      0 SNES Function norm 1.132185384796e-08 
>>        0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>        1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>      1 SNES Function norm 2.177599365111e-12 
>>    Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>    0 SNES Function norm 5.066222213176e+03 
>>      0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>    1 SNES Function norm 8.482593852817e+02 
>>      0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>      2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>    2 SNES Function norm 6.543140468549e+02 
>>      0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>      2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>    3 SNES Function norm 5.766430557220e+02 
>>      0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>      2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>    4 SNES Function norm 5.235211958260e+02 
>>      0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>      2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>    5 SNES Function norm 4.752913229649e+02 
>>      0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>      2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>    6 SNES Function norm 4.220255380391e+02 
>>      0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>      2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>    7 SNES Function norm 3.805408907074e+02 
>>      0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>    8 SNES Function norm 3.764619752339e+02 
>>      0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>      2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>    9 SNES Function norm 3.761182227091e+02 
>>      0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>      2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>   10 SNES Function norm 3.740017190063e+02 
>>      0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>      2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>   11 SNES Function norm 3.725903477238e+02 
>>      0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>      2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>   12 SNES Function norm 3.716162097231e+02 
>>      0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>      2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>   13 SNES Function norm 3.674168632847e+02 
>>      0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>      2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>   14 SNES Function norm 3.532395445266e+02 
>>      0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>      2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>   15 SNES Function norm 3.182438872366e+02 
>>      0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>      2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>   16 SNES Function norm 3.091759892779e+02 
>>      0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>      2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>   17 SNES Function norm 2.987839504359e+02 
>>      0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>      2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>   18 SNES Function norm 2.987073622777e+02 
>>      0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>      2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>   19 SNES Function norm 2.987067936734e+02 
>>      0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>      2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>   20 SNES Function norm 2.987067502910e+02 
>>      0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>      2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>   21 SNES Function norm 2.987064584431e+02 
>>      0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>      2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>   22 SNES Function norm 2.987064525262e+02 
>>      0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>      2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>   23 SNES Function norm 2.987064121622e+02 
>>      0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>      2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>   24 SNES Function norm 2.987063973426e+02 
>>      0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>      2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>   25 SNES Function norm 2.987063920553e+02 
>>      0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>      2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>   26 SNES Function norm 2.987063919786e+02 
>>      0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>      2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>   27 SNES Function norm 2.987063393145e+02 
>>      0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>      2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>   28 SNES Function norm 2.987063253277e+02 
>>      0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>      2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>   29 SNES Function norm 2.987063197153e+02 
>>      0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>      2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>   30 SNES Function norm 2.987063193089e+02 
>>      0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>      2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>   31 SNES Function norm 2.987063192061e+02 
>>      0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>      2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>   32 SNES Function norm 2.987063094492e+02 
>>      0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>      2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>   33 SNES Function norm 2.987063089942e+02 
>>      0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>      2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>   34 SNES Function norm 2.987062660109e+02 
>>      0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>      2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>   35 SNES Function norm 2.987062107900e+02 
>>      0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>      2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>   36 SNES Function norm 2.987062055224e+02 
>>      0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>      2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>   37 SNES Function norm 2.987061774798e+02 
>>      0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>      2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>   38 SNES Function norm 2.987061715400e+02 
>>      0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>      2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>   39 SNES Function norm 2.987061699634e+02 
>>      0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>      2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>   40 SNES Function norm 2.987061630064e+02 
>>      0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>   41 SNES Function norm 2.987061630064e+02 
>>      0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>   42 SNES Function norm 2.987061630064e+02 
>>      0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>   43 SNES Function norm 2.987061630064e+02 
>>      0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>   44 SNES Function norm 2.987061630064e+02 
>>      0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>   45 SNES Function norm 2.987061630064e+02 
>>      0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>   46 SNES Function norm 2.987061630064e+02 
>>      0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>   47 SNES Function norm 2.987061630064e+02 
>>      0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>  Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>> 
>> 
>> -gideon
>> 
>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>> 
>>>  ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>> 
>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>> 
>>> This is a minor glitch we'll get past.
>>> 
>>> Barry
>>> 
>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>> 
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>> [0]PETSC ERROR: Mat type mffd
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> Hmm,
>>>>> 
>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>> 
>>>>> gdb ./blowup_batch_refine
>>>>> 
>>>>> then 
>>>>> 
>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>> 
>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>> 
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> Running with that flag gives me this:
>>>>>> 
>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>> Unable to start debugger: No such file or directory
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>> 
>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>> 
>>>>>>>>>>  0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>  1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>> type: newtonls
>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>> type: bt
>>>>>>>>>>  interpolation: cubic
>>>>>>>>>>  alpha=1.000000e-04
>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>> maximum iterations=40
>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>> type: gmres
>>>>>>>>>>  GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>  GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>> left preconditioning
>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>> type: lu
>>>>>>>>>>  LU: out-of-place factorization
>>>>>>>>>>  tolerance for zero pivot 2.22045e-14
>>>>>>>>>>  matrix ordering: nd
>>>>>>>>>>  factor fill ratio given 0, needed 0
>>>>>>>>>>    Factored matrix follows:
>>>>>>>>>>      Mat Object:           1 MPI processes
>>>>>>>>>>        type: seqaij
>>>>>>>>>>        rows=15991, cols=15991
>>>>>>>>>>        package used to perform factorization: mumps
>>>>>>>>>>        total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>        total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>          MUMPS run parameters:
>>>>>>>>>>            SYM (matrix type):                   0 
>>>>>>>>>>            PAR (host participation):            1 
>>>>>>>>>>            ICNTL(1) (output for error):         6 
>>>>>>>>>>            ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>            ICNTL(3) (output for global info):   0 
>>>>>>>>>>            ICNTL(4) (level of printing):        0 
>>>>>>>>>>            ICNTL(5) (input mat struct):         0 
>>>>>>>>>>            ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>            ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>            ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>            ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>            ICNTL(11) (error analysis):          0 
>>>>>>>>>>            ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>            ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>            ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>            ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>            ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>            ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>            ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>            ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>            ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>            ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>            ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>            ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>            ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>            ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>            ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>            ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>            ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>            ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>            CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>            CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>            CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>            CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>            CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>            RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>              [0] 1.95838e+06 
>>>>>>>>>>            RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>              [0]  143924 
>>>>>>>>>>            RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>              [0]  1.95943e+06 
>>>>>>>>>>            INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>            [0] 7 
>>>>>>>>>>            INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>              [0] 7 
>>>>>>>>>>            INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>              [0] 15991 
>>>>>>>>>>            RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>            RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>            RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>            (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>            INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>            INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>            INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>            INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>            INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>            INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>            INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>            INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>            INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>            INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>            INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>            INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>            INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>            INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>            INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>            INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>            INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>            INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>            INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>            INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>            INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>            INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>            INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>            INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>            INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>            INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>            INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>            INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>            INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>  type: seqaij
>>>>>>>>>>  rows=15991, cols=15991
>>>>>>>>>>  total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>  total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>    using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> Barry,
>>>>>>>>>>> 
>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>> 
>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>> 
>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Matt
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>> 
>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>> 
>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>> 
>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>> 
>>>>>>>>>>>>   SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>> 
>>>>>>>>>>>>   if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>        SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>        SNESSetDM()
>>>>>>>>>>>>        etc
>>>>>>>>>>>>        SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>        SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>        SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>        VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>        SNESDestroy(snesrefine);
>>>>>>>>>>>>  end if
>>>>>>>>>>>> 
>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -- 
>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150907/3308c4be/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon Sep  7 21:27:34 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Sep 2015 21:27:34 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
Message-ID: <4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>


  If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.

  You can try -snes_mf -pc_type none and see how the linear solver converges. 


  Barry



> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
> 
> When I run with -ksp_type fgmres, I get
> 
>       0 SNES Function norm 3.857327250267e-09 
>         0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>         1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>         2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>       1 SNES Function norm 2.479697868412e-12 
>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>     0 SNES Function norm 5.066222213176e+03 
>       0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>     1 SNES Function norm 8.484696862318e+02 
>       0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>       2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>     2 SNES Function norm 6.551139809777e+02 
>       0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>       2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>     3 SNES Function norm 5.771483337231e+02 
>       0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>       2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>     4 SNES Function norm 5.237540245670e+02 
>       0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>       2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>     5 SNES Function norm 4.753058321803e+02 
>       0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>       2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>     6 SNES Function norm 4.221482124411e+02 
>       0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>       2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>     7 SNES Function norm 3.803620028964e+02 
>       0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>       2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>     8 SNES Function norm 3.769519577570e+02 
>       0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>       2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>       3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>     9 SNES Function norm 3.766389203406e+02 
>       0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>       2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>    10 SNES Function norm 3.751822489648e+02 
>       0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>       2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>    11 SNES Function norm 3.737189963925e+02 
>       0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>       2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>       3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>    12 SNES Function norm 3.737110331594e+02 
>       0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>       2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>    13 SNES Function norm 3.684175513154e+02 
>       0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>       2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>    14 SNES Function norm 3.509930783933e+02 
>       0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>       2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>    15 SNES Function norm 3.162121096565e+02 
>       0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>       2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>    16 SNES Function norm 2.890165052818e+02 
>       0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>       2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>    17 SNES Function norm 2.816929421445e+02 
>       0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>       2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>       3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>    18 SNES Function norm 2.814407559738e+02 
>       0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>       2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>    19 SNES Function norm 2.810261821835e+02 
>       0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>       2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>    20 SNES Function norm 2.755723672077e+02 
>       0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>       2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>       3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>       4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>       5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>       6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>    21 SNES Function norm 2.755720389981e+02 
>       0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>       2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>       3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>       4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>       5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>       6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>    22 SNES Function norm 2.755716464881e+02 
>       0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>       2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>       3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>       4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>    23 SNES Function norm 2.755713163168e+02 
>       0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>       2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>       3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>       4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>       5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>       6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>       7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>       8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>    24 SNES Function norm 2.755712363532e+02 
>       0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>       2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>       3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>       4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>       5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>       6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>       7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>       8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>       9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>      10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>      11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>      12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>      13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>      14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>      15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>      16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>      17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>      18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>      19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>      20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>      21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>      22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>      23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>      24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>      25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>      26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>      27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>      28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>      29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>      30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>      31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>      32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>      33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>      34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>      35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>      36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>      37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>      38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>      39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>      40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>      41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>      42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>      43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>      44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>      45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>      46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>      47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>      48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>      49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>      50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>      51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>      52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>      53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>      54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>      55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>      56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>      57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>      58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>      59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>      60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>      61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>      62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>      63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>      64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>      65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>      66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>      67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>      68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>      69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>      70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>      71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>      72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>      73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>      74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>      75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>      76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>      77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>      78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>      79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>      80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>      81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>      82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>      83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>      84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>      85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>      86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>      87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>      88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>      89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>      90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>      91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>      92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>      93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>      94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>      95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>      96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>      97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>      98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>      99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>     100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>     101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>     102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>     103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>     104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>     105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>     106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>     107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>     108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>     109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>     110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>     111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>     112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>     113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>     114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>     115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>     116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>     117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>     118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>     119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>     120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>     121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>     122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>     123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>     124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>     125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>     126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>     127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>     128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>     129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>     130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>     131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>     132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>     133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>     134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>     135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>     136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>     137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>     138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>     139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>     140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>     141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>     142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>     143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>     144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>     145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>     146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>     147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>     148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>     149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>     150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>     151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>     152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>     153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>     154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>     155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>     156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>     157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>     158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>     159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>     160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>     161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>     162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>     163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>     164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>     165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>     166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>     167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>     168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>     169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>     170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>     171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>     172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>     173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>     174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>     175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>     176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>     177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>     178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>     179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>     180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>     181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>     182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>     183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>     184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>     185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>     186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>     187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>     188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>     189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>     190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>     191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>     192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>     193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>     194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>     195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>     196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>     197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>     198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>     199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>     200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>     201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>     202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>     203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>     204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>     205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>     206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>     207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>     208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>     209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>     210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
> 
> -gideon
> 
>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>> 
>>  If not run with with -ksp_type fgmres and send the same output.
>> 
>>  Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>> 
>>  Barry
>> 
>> 
>>   21 SNES Function norm 2.987064584431e+02 
>>      0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>      2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>> 
>> 
>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>> 
>>>      0 SNES Function norm 1.132185384796e-08 
>>>        0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>        1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>      1 SNES Function norm 2.177599365111e-12 
>>>    Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>    0 SNES Function norm 5.066222213176e+03 
>>>      0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>    1 SNES Function norm 8.482593852817e+02 
>>>      0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>      2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>    2 SNES Function norm 6.543140468549e+02 
>>>      0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>      2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>    3 SNES Function norm 5.766430557220e+02 
>>>      0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>      2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>    4 SNES Function norm 5.235211958260e+02 
>>>      0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>      2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>    5 SNES Function norm 4.752913229649e+02 
>>>      0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>      2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>    6 SNES Function norm 4.220255380391e+02 
>>>      0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>      2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>    7 SNES Function norm 3.805408907074e+02 
>>>      0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>    8 SNES Function norm 3.764619752339e+02 
>>>      0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>      2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>    9 SNES Function norm 3.761182227091e+02 
>>>      0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>      2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>   10 SNES Function norm 3.740017190063e+02 
>>>      0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>      2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>   11 SNES Function norm 3.725903477238e+02 
>>>      0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>      2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>   12 SNES Function norm 3.716162097231e+02 
>>>      0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>      2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>   13 SNES Function norm 3.674168632847e+02 
>>>      0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>      2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>   14 SNES Function norm 3.532395445266e+02 
>>>      0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>      2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>   15 SNES Function norm 3.182438872366e+02 
>>>      0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>      2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>   16 SNES Function norm 3.091759892779e+02 
>>>      0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>      2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>   17 SNES Function norm 2.987839504359e+02 
>>>      0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>      2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>   18 SNES Function norm 2.987073622777e+02 
>>>      0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>      2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>   19 SNES Function norm 2.987067936734e+02 
>>>      0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>      2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>   20 SNES Function norm 2.987067502910e+02 
>>>      0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>      2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>   21 SNES Function norm 2.987064584431e+02 
>>>      0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>      2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>   22 SNES Function norm 2.987064525262e+02 
>>>      0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>      2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>   23 SNES Function norm 2.987064121622e+02 
>>>      0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>      2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>   24 SNES Function norm 2.987063973426e+02 
>>>      0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>      2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>   25 SNES Function norm 2.987063920553e+02 
>>>      0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>      2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>   26 SNES Function norm 2.987063919786e+02 
>>>      0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>      2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>   27 SNES Function norm 2.987063393145e+02 
>>>      0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>      2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>   28 SNES Function norm 2.987063253277e+02 
>>>      0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>      2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>   29 SNES Function norm 2.987063197153e+02 
>>>      0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>      2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>   30 SNES Function norm 2.987063193089e+02 
>>>      0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>      2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>   31 SNES Function norm 2.987063192061e+02 
>>>      0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>      2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>   32 SNES Function norm 2.987063094492e+02 
>>>      0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>      2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>   33 SNES Function norm 2.987063089942e+02 
>>>      0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>      2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>   34 SNES Function norm 2.987062660109e+02 
>>>      0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>      2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>   35 SNES Function norm 2.987062107900e+02 
>>>      0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>      2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>   36 SNES Function norm 2.987062055224e+02 
>>>      0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>      2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>   37 SNES Function norm 2.987061774798e+02 
>>>      0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>      2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>   38 SNES Function norm 2.987061715400e+02 
>>>      0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>      2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>   39 SNES Function norm 2.987061699634e+02 
>>>      0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>      2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>   40 SNES Function norm 2.987061630064e+02 
>>>      0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>   41 SNES Function norm 2.987061630064e+02 
>>>      0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>   42 SNES Function norm 2.987061630064e+02 
>>>      0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>   43 SNES Function norm 2.987061630064e+02 
>>>      0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>   44 SNES Function norm 2.987061630064e+02 
>>>      0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>   45 SNES Function norm 2.987061630064e+02 
>>>      0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>   46 SNES Function norm 2.987061630064e+02 
>>>      0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>   47 SNES Function norm 2.987061630064e+02 
>>>      0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>  Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>> 
>>> 
>>> -gideon
>>> 
>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>> 
>>>>  ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>> 
>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>> 
>>>> This is a minor glitch we'll get past.
>>>> 
>>>> Barry
>>>> 
>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>> 
>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>> [0]PETSC ERROR: Mat type mffd
>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> Hmm,
>>>>>> 
>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>> 
>>>>>> gdb ./blowup_batch_refine
>>>>>> 
>>>>>> then 
>>>>>> 
>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>> 
>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>> 
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> Running with that flag gives me this:
>>>>>>> 
>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>> Unable to start debugger: No such file or directory
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>> 
>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>> 
>>>>>>>>>>>  0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>  1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>> type: newtonls
>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>> type: bt
>>>>>>>>>>>  interpolation: cubic
>>>>>>>>>>>  alpha=1.000000e-04
>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>> type: gmres
>>>>>>>>>>>  GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>  GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>> left preconditioning
>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>> type: lu
>>>>>>>>>>>  LU: out-of-place factorization
>>>>>>>>>>>  tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>  matrix ordering: nd
>>>>>>>>>>>  factor fill ratio given 0, needed 0
>>>>>>>>>>>    Factored matrix follows:
>>>>>>>>>>>      Mat Object:           1 MPI processes
>>>>>>>>>>>        type: seqaij
>>>>>>>>>>>        rows=15991, cols=15991
>>>>>>>>>>>        package used to perform factorization: mumps
>>>>>>>>>>>        total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>        total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>          MUMPS run parameters:
>>>>>>>>>>>            SYM (matrix type):                   0 
>>>>>>>>>>>            PAR (host participation):            1 
>>>>>>>>>>>            ICNTL(1) (output for error):         6 
>>>>>>>>>>>            ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>            ICNTL(3) (output for global info):   0 
>>>>>>>>>>>            ICNTL(4) (level of printing):        0 
>>>>>>>>>>>            ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>            ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>            ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>            ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>            ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>            ICNTL(11) (error analysis):          0 
>>>>>>>>>>>            ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>            ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>            ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>            ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>            ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>            ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>            ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>            ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>            ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>            ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>            ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>            ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>            ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>            ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>            ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>            ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>            ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>            ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>            CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>            CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>            CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>            CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>            CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>            RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>              [0] 1.95838e+06 
>>>>>>>>>>>            RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>              [0]  143924 
>>>>>>>>>>>            RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>              [0]  1.95943e+06 
>>>>>>>>>>>            INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>            [0] 7 
>>>>>>>>>>>            INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>              [0] 7 
>>>>>>>>>>>            INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>              [0] 15991 
>>>>>>>>>>>            RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>            RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>            RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>            (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>            INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>            INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>            INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>            INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>            INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>            INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>            INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>            INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>            INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>            INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>            INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>            INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>            INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>            INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>            INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>            INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>            INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>            INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>            INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>            INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>            INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>            INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>            INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>            INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>            INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>            INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>            INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>            INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>            INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>  type: seqaij
>>>>>>>>>>>  rows=15991, cols=15991
>>>>>>>>>>>  total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>  total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>    using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>> Barry,
>>>>>>>>>>>> 
>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>> 
>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>> 
>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> 
>>>>>>>>>>>> Matt
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>> 
>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>> 
>>>>>>>>>>>>>   SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>> 
>>>>>>>>>>>>>   if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>        SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>        SNESSetDM()
>>>>>>>>>>>>>        etc
>>>>>>>>>>>>>        SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>        SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>        SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>        VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>        SNESDestroy(snesrefine);
>>>>>>>>>>>>>  end if
>>>>>>>>>>>>> 
>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -- 
>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From gideon.simpson at gmail.com  Mon Sep  7 21:35:34 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 7 Sep 2015 22:35:34 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
Message-ID: <75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>

Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
 
SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);


-gideon

> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
> 
>  You can try -snes_mf -pc_type none and see how the linear solver converges. 
> 
> 
>  Barry
> 
> 
> 
>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>> 
>> When I run with -ksp_type fgmres, I get
>> 
>>      0 SNES Function norm 3.857327250267e-09 
>>        0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>        1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>        2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>      1 SNES Function norm 2.479697868412e-12 
>>    Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>    0 SNES Function norm 5.066222213176e+03 
>>      0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>    1 SNES Function norm 8.484696862318e+02 
>>      0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>      2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>    2 SNES Function norm 6.551139809777e+02 
>>      0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>      2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>    3 SNES Function norm 5.771483337231e+02 
>>      0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>      2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>    4 SNES Function norm 5.237540245670e+02 
>>      0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>      2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>    5 SNES Function norm 4.753058321803e+02 
>>      0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>      2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>    6 SNES Function norm 4.221482124411e+02 
>>      0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>      2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>    7 SNES Function norm 3.803620028964e+02 
>>      0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>      2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>    8 SNES Function norm 3.769519577570e+02 
>>      0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>      2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>      3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>    9 SNES Function norm 3.766389203406e+02 
>>      0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>      2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>   10 SNES Function norm 3.751822489648e+02 
>>      0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>      2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>   11 SNES Function norm 3.737189963925e+02 
>>      0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>      2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>      3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>   12 SNES Function norm 3.737110331594e+02 
>>      0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>      2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>   13 SNES Function norm 3.684175513154e+02 
>>      0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>      2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>   14 SNES Function norm 3.509930783933e+02 
>>      0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>      2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>   15 SNES Function norm 3.162121096565e+02 
>>      0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>      2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>   16 SNES Function norm 2.890165052818e+02 
>>      0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>      2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>   17 SNES Function norm 2.816929421445e+02 
>>      0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>      2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>      3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>   18 SNES Function norm 2.814407559738e+02 
>>      0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>      2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>   19 SNES Function norm 2.810261821835e+02 
>>      0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>      2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>   20 SNES Function norm 2.755723672077e+02 
>>      0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>      2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>      3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>      4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>      5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>      6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>   21 SNES Function norm 2.755720389981e+02 
>>      0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>      2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>      3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>      4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>      5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>      6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>   22 SNES Function norm 2.755716464881e+02 
>>      0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>      2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>      3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>      4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>   23 SNES Function norm 2.755713163168e+02 
>>      0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>      2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>      3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>      4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>      5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>      6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>      7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>      8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>   24 SNES Function norm 2.755712363532e+02 
>>      0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>      2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>      3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>      4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>      5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>      6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>      7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>      8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>      9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>     10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>     11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>     12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>     13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>     14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>     15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>     16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>     17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>     18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>     19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>     20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>     21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>     22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>     23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>     24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>     25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>     26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>     27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>     28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>     29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>     30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>     31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>     32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>     33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>     34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>     35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>     36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>     37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>     38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>     39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>     40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>     41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>     42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>     43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>     44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>     45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>     46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>     47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>     48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>     49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>     50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>     51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>     52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>     53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>     54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>     55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>     56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>     57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>     58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>     59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>     60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>     61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>     62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>     63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>     64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>     65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>     66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>     67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>     68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>     69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>     70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>     71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>     72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>     73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>     74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>     75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>     76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>     77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>     78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>     79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>     80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>     81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>     82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>     83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>     84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>     85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>     86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>     87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>     88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>     89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>     90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>     91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>     92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>     93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>     94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>     95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>     96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>     97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>     98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>     99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>    100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>    101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>    102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>    103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>    104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>    105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>    106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>    107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>    108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>    109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>    110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>    111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>    112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>    113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>    114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>    115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>    116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>    117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>    118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>    119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>    120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>    121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>    122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>    123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>    124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>    125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>    126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>    127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>    128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>    129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>    130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>    131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>    132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>    133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>    134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>    135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>    136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>    137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>    138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>    139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>    140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>    141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>    142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>    143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>    144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>    145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>    146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>    147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>    148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>    149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>    150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>    151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>    152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>    153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>    154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>    155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>    156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>    157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>    158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>    159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>    160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>    161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>    162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>    163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>    164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>    165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>    166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>    167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>    168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>    169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>    170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>    171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>    172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>    173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>    174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>    175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>    176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>    177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>    178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>    179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>    180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>    181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>    182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>    183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>    184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>    185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>    186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>    187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>    188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>    189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>    190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>    191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>    192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>    193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>    194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>    195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>    196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>    197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>    198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>    199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>    200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>    201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>    202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>    203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>    204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>    205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>    206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>    207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>    208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>    209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>    210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>  Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>> 
>> -gideon
>> 
>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>> 
>>> If not run with with -ksp_type fgmres and send the same output.
>>> 
>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>> 
>>> Barry
>>> 
>>> 
>>>  21 SNES Function norm 2.987064584431e+02 
>>>     0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>     2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>> 
>>> 
>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>> 
>>>>     0 SNES Function norm 1.132185384796e-08 
>>>>       0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>       1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>     1 SNES Function norm 2.177599365111e-12 
>>>>   Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>   0 SNES Function norm 5.066222213176e+03 
>>>>     0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>   1 SNES Function norm 8.482593852817e+02 
>>>>     0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>     2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>   2 SNES Function norm 6.543140468549e+02 
>>>>     0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>     2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>   3 SNES Function norm 5.766430557220e+02 
>>>>     0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>     2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>   4 SNES Function norm 5.235211958260e+02 
>>>>     0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>     2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>   5 SNES Function norm 4.752913229649e+02 
>>>>     0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>     2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>   6 SNES Function norm 4.220255380391e+02 
>>>>     0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>     2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>   7 SNES Function norm 3.805408907074e+02 
>>>>     0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>   8 SNES Function norm 3.764619752339e+02 
>>>>     0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>     2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>   9 SNES Function norm 3.761182227091e+02 
>>>>     0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>     2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>  10 SNES Function norm 3.740017190063e+02 
>>>>     0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>     2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>  11 SNES Function norm 3.725903477238e+02 
>>>>     0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>     2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>  12 SNES Function norm 3.716162097231e+02 
>>>>     0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>     2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>  13 SNES Function norm 3.674168632847e+02 
>>>>     0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>     2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>  14 SNES Function norm 3.532395445266e+02 
>>>>     0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>     2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>  15 SNES Function norm 3.182438872366e+02 
>>>>     0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>     2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>  16 SNES Function norm 3.091759892779e+02 
>>>>     0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>     2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>  17 SNES Function norm 2.987839504359e+02 
>>>>     0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>     2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>  18 SNES Function norm 2.987073622777e+02 
>>>>     0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>     2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>  19 SNES Function norm 2.987067936734e+02 
>>>>     0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>     2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>  20 SNES Function norm 2.987067502910e+02 
>>>>     0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>     2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>  21 SNES Function norm 2.987064584431e+02 
>>>>     0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>     2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>  22 SNES Function norm 2.987064525262e+02 
>>>>     0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>     2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>  23 SNES Function norm 2.987064121622e+02 
>>>>     0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>     2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>  24 SNES Function norm 2.987063973426e+02 
>>>>     0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>     2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>  25 SNES Function norm 2.987063920553e+02 
>>>>     0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>     2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>  26 SNES Function norm 2.987063919786e+02 
>>>>     0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>     2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>  27 SNES Function norm 2.987063393145e+02 
>>>>     0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>     2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>  28 SNES Function norm 2.987063253277e+02 
>>>>     0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>     2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>  29 SNES Function norm 2.987063197153e+02 
>>>>     0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>     2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>  30 SNES Function norm 2.987063193089e+02 
>>>>     0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>     2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>  31 SNES Function norm 2.987063192061e+02 
>>>>     0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>     2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>  32 SNES Function norm 2.987063094492e+02 
>>>>     0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>     2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>  33 SNES Function norm 2.987063089942e+02 
>>>>     0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>     2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>  34 SNES Function norm 2.987062660109e+02 
>>>>     0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>     2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>  35 SNES Function norm 2.987062107900e+02 
>>>>     0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>     2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>  36 SNES Function norm 2.987062055224e+02 
>>>>     0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>     2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>  37 SNES Function norm 2.987061774798e+02 
>>>>     0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>     2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>  38 SNES Function norm 2.987061715400e+02 
>>>>     0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>     2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>  39 SNES Function norm 2.987061699634e+02 
>>>>     0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>     2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>  40 SNES Function norm 2.987061630064e+02 
>>>>     0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>  41 SNES Function norm 2.987061630064e+02 
>>>>     0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>  42 SNES Function norm 2.987061630064e+02 
>>>>     0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>  43 SNES Function norm 2.987061630064e+02 
>>>>     0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>  44 SNES Function norm 2.987061630064e+02 
>>>>     0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>  45 SNES Function norm 2.987061630064e+02 
>>>>     0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>  46 SNES Function norm 2.987061630064e+02 
>>>>     0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>  47 SNES Function norm 2.987061630064e+02 
>>>>     0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>> 
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>> 
>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>> 
>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>> 
>>>>> This is a minor glitch we'll get past.
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>> 
>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> Hmm,
>>>>>>> 
>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>> 
>>>>>>> gdb ./blowup_batch_refine
>>>>>>> 
>>>>>>> then 
>>>>>>> 
>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>> 
>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>> 
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Running with that flag gives me this:
>>>>>>>> 
>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>> 
>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>> 
>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>> type: bt
>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>> type: gmres
>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>> type: lu
>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>   Factored matrix follows:
>>>>>>>>>>>>     Mat Object:           1 MPI processes
>>>>>>>>>>>>       type: seqaij
>>>>>>>>>>>>       rows=15991, cols=15991
>>>>>>>>>>>>       package used to perform factorization: mumps
>>>>>>>>>>>>       total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>       total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>         MUMPS run parameters:
>>>>>>>>>>>>           SYM (matrix type):                   0 
>>>>>>>>>>>>           PAR (host participation):            1 
>>>>>>>>>>>>           ICNTL(1) (output for error):         6 
>>>>>>>>>>>>           ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>           ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>           ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>           ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>           ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>           ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>           ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>           ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>           ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>           ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>           ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>           ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>           ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>           ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>           ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>           ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>           ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>           ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>           ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>           ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>           ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>           ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>           ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>           ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>           ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>           ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>           ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>           CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>           CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>           CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>           CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>           CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>           RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>             [0] 1.95838e+06 
>>>>>>>>>>>>           RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>             [0]  143924 
>>>>>>>>>>>>           RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>             [0]  1.95943e+06 
>>>>>>>>>>>>           INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>           [0] 7 
>>>>>>>>>>>>           INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>             [0] 7 
>>>>>>>>>>>>           INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>             [0] 15991 
>>>>>>>>>>>>           RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>           RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>           RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>           (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>           INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>           INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>           INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>           INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>           INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>           INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>           INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>           INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>           INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>           INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>           INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>           INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>           INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>           INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>           INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>           INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>           INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>           INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>           INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>           INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>           INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>           INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>           INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>           INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>           INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>           INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>           INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>           INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>           INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>   using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Matt
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>  SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>  if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>       SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>       SNESSetDM()
>>>>>>>>>>>>>>       etc
>>>>>>>>>>>>>>       SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>       SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>       SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>       VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>       SNESDestroy(snesrefine);
>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From bsmith at mcs.anl.gov  Mon Sep  7 21:58:08 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Sep 2015 21:58:08 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
Message-ID: <30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>


> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
> 
> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);

Gideon,

   In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none

  Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct


  Barry

> 
> 
> -gideon
> 
>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>> 
>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>> 
>> 
>> Barry
>> 
>> 
>> 
>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>> 
>>> When I run with -ksp_type fgmres, I get
>>> 
>>>     0 SNES Function norm 3.857327250267e-09 
>>>       0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>       2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>     1 SNES Function norm 2.479697868412e-12 
>>>   Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>   0 SNES Function norm 5.066222213176e+03 
>>>     0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>   1 SNES Function norm 8.484696862318e+02 
>>>     0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>     2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>   2 SNES Function norm 6.551139809777e+02 
>>>     0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>     2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>   3 SNES Function norm 5.771483337231e+02 
>>>     0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>     2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>   4 SNES Function norm 5.237540245670e+02 
>>>     0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>     2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>   5 SNES Function norm 4.753058321803e+02 
>>>     0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>     2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>   6 SNES Function norm 4.221482124411e+02 
>>>     0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>     2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>   7 SNES Function norm 3.803620028964e+02 
>>>     0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>     2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>   8 SNES Function norm 3.769519577570e+02 
>>>     0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>     2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>     3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>   9 SNES Function norm 3.766389203406e+02 
>>>     0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>     2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>  10 SNES Function norm 3.751822489648e+02 
>>>     0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>     2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>  11 SNES Function norm 3.737189963925e+02 
>>>     0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>     2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>     3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>  12 SNES Function norm 3.737110331594e+02 
>>>     0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>     2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>  13 SNES Function norm 3.684175513154e+02 
>>>     0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>     2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>  14 SNES Function norm 3.509930783933e+02 
>>>     0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>     2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>  15 SNES Function norm 3.162121096565e+02 
>>>     0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>     2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>  16 SNES Function norm 2.890165052818e+02 
>>>     0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>     2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>  17 SNES Function norm 2.816929421445e+02 
>>>     0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>     2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>     3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>  18 SNES Function norm 2.814407559738e+02 
>>>     0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>     2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>  19 SNES Function norm 2.810261821835e+02 
>>>     0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>     2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>  20 SNES Function norm 2.755723672077e+02 
>>>     0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>     2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>     3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>     4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>     5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>     6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>  21 SNES Function norm 2.755720389981e+02 
>>>     0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>     2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>     3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>     4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>     5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>     6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>  22 SNES Function norm 2.755716464881e+02 
>>>     0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>     2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>     3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>     4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>  23 SNES Function norm 2.755713163168e+02 
>>>     0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>     2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>     3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>     4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>     5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>     6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>     7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>     8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>  24 SNES Function norm 2.755712363532e+02 
>>>     0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>     2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>     3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>     4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>     5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>     6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>     7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>     8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>     9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>    10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>    11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>    12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>    13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>    14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>    15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>    16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>    17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>    18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>    19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>    20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>    21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>    22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>    23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>    24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>    25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>    26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>    27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>    28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>    29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>    30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>    31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>    32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>    33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>    34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>    35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>    36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>    37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>    38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>    39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>    40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>    41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>    42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>    43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>    44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>    45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>    46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>    47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>    48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>    49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>    50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>    51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>    52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>    53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>    54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>    55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>    56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>    57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>    58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>    59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>    60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>    61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>    62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>    63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>    64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>    65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>    66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>    67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>    68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>    69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>    70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>    71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>    72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>    73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>    74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>    75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>    76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>    77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>    78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>    79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>    80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>    81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>    82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>    83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>    84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>    85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>    86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>    87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>    88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>    89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>    90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>    91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>    92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>    93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>    94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>    95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>    96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>    97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>    98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>    99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>   100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>   101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>   102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>   103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>   104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>   105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>   106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>   107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>   108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>   109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>   110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>   111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>   112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>   113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>   114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>   115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>   116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>   117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>   118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>   119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>   120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>   121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>   122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>   123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>   124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>   125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>   126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>   127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>   128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>   129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>   130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>   131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>   132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>   133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>   134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>   135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>   136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>   137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>   138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>   139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>   140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>   141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>   142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>   143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>   144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>   145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>   146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>   147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>   148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>   149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>   150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>   151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>   152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>   153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>   154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>   155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>   156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>   157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>   158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>   159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>   160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>   161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>   162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>   163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>   164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>   165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>   166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>   167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>   168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>   169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>   170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>   171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>   172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>   173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>   174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>   175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>   176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>   177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>   178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>   179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>   180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>   181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>   182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>   183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>   184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>   185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>   186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>   187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>   188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>   189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>   190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>   191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>   192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>   193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>   194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>   195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>   196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>   197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>   198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>   199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>   200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>   201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>   202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>   203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>   204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>   205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>   206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>   207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>   208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>   209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>   210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>> 
>>> -gideon
>>> 
>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>> 
>>>> If not run with with -ksp_type fgmres and send the same output.
>>>> 
>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>    0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>    2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>> 
>>>> 
>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>> 
>>>>>    0 SNES Function norm 1.132185384796e-08 
>>>>>      0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>      1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>    1 SNES Function norm 2.177599365111e-12 
>>>>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>  0 SNES Function norm 5.066222213176e+03 
>>>>>    0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>  1 SNES Function norm 8.482593852817e+02 
>>>>>    0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>    2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>  2 SNES Function norm 6.543140468549e+02 
>>>>>    0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>    2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>  3 SNES Function norm 5.766430557220e+02 
>>>>>    0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>    2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>  4 SNES Function norm 5.235211958260e+02 
>>>>>    0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>    2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>  5 SNES Function norm 4.752913229649e+02 
>>>>>    0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>    2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>  6 SNES Function norm 4.220255380391e+02 
>>>>>    0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>    2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>  7 SNES Function norm 3.805408907074e+02 
>>>>>    0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>  8 SNES Function norm 3.764619752339e+02 
>>>>>    0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>    2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>  9 SNES Function norm 3.761182227091e+02 
>>>>>    0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>    2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>    0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>    2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>    0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>    2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>    0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>    2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>    0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>    2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>    0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>    2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>    0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>    2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>    0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>    2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>    0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>    2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>    0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>    2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>    0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>    2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>    0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>    2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>    0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>    2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>    0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>    2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>    0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>    2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>    0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>    2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>    0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>    2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>    0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>    2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>    0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>    2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>    0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>    2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>    0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>    2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>    0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>    2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>    0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>    2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>    0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>    2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>    0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>    2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>    0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>    2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>    0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>    2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>    0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>    2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>    0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>    2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>    0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>    2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>    0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>    2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>    0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>    0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>    0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>    0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>    0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>    0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>    0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>    0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>> 
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>> 
>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>> 
>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>> 
>>>>>> This is a minor glitch we'll get past.
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>> 
>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Hmm,
>>>>>>>> 
>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>> 
>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>> 
>>>>>>>> then 
>>>>>>>> 
>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>> 
>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Running with that flag gives me this:
>>>>>>>>> 
>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>> 
>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>  Factored matrix follows:
>>>>>>>>>>>>>    Mat Object:           1 MPI processes
>>>>>>>>>>>>>      type: seqaij
>>>>>>>>>>>>>      rows=15991, cols=15991
>>>>>>>>>>>>>      package used to perform factorization: mumps
>>>>>>>>>>>>>      total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>      total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>        MUMPS run parameters:
>>>>>>>>>>>>>          SYM (matrix type):                   0 
>>>>>>>>>>>>>          PAR (host participation):            1 
>>>>>>>>>>>>>          ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>          ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>          ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>          ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>          ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>          ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>          ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>          ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>          ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>          ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>          ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>          ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>          ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>          ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>          ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>          ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>          ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>          ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>          ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>          ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>          ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>          ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>          ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>          ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>          ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>          ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>          ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>          ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>          CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>          CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>          CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>          CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>          CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>          RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>            [0] 1.95838e+06 
>>>>>>>>>>>>>          RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>            [0]  143924 
>>>>>>>>>>>>>          RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>            [0]  1.95943e+06 
>>>>>>>>>>>>>          INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>          INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>            [0] 7 
>>>>>>>>>>>>>          INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>            [0] 15991 
>>>>>>>>>>>>>          RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>          RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>          RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>          (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>          INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>          INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>          INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>          INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>          INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>          INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>          INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>          INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>          INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>          INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>          INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>          INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>          INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>          INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>          INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>          INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>          INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>          INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>          INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>          INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>          INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>          INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>          INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>          INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>          INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>          INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>          INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>          INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>          INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>  using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>      SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>      SNESSetDM()
>>>>>>>>>>>>>>>      etc
>>>>>>>>>>>>>>>      SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>      SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>      SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>      VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>      SNESDestroy(snesrefine);
>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From gideon.simpson at gmail.com  Tue Sep  8 10:14:28 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 11:14:28 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
Message-ID: <2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>

Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.


-gideon

> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>> 
>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
> 
> Gideon,
> 
>   In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
> 
>  Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
> 
> 
>  Barry
> 
>> 
>> 
>> -gideon
>> 
>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>> 
>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>> 
>>> 
>>> Barry
>>> 
>>> 
>>> 
>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>> 
>>>> When I run with -ksp_type fgmres, I get
>>>> 
>>>>    0 SNES Function norm 3.857327250267e-09 
>>>>      0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>      1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>      2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>    1 SNES Function norm 2.479697868412e-12 
>>>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>  0 SNES Function norm 5.066222213176e+03 
>>>>    0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>  1 SNES Function norm 8.484696862318e+02 
>>>>    0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>    2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>  2 SNES Function norm 6.551139809777e+02 
>>>>    0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>    2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>  3 SNES Function norm 5.771483337231e+02 
>>>>    0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>    2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>  4 SNES Function norm 5.237540245670e+02 
>>>>    0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>    2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>  5 SNES Function norm 4.753058321803e+02 
>>>>    0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>    2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>  6 SNES Function norm 4.221482124411e+02 
>>>>    0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>    2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>  7 SNES Function norm 3.803620028964e+02 
>>>>    0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>    2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>  8 SNES Function norm 3.769519577570e+02 
>>>>    0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>    2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>    3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>  9 SNES Function norm 3.766389203406e+02 
>>>>    0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>    2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>    0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>    2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>    0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>    2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>    3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>    0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>    2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>    0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>    2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>    0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>    2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>    0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>    2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>    0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>    2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>    0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>    2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>    3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>    0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>    2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>    0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>    2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>    0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>    2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>    3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>    4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>    5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>    6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>    0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>    2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>    3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>    4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>    5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>    6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>    0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>    2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>    3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>    4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>    0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>    2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>    3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>    4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>    5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>    6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>    7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>    8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>    0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>    1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>    2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>    3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>    4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>    5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>    6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>    7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>    8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>    9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>   10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>   11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>   12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>   13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>   14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>   15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>   16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>   17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>   18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>   19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>   20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>   21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>   22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>   23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>   24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>   25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>   26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>   27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>   28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>   29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>   30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>   31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>   32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>   33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>   34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>   35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>   36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>   37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>   38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>   39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>   40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>   41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>   42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>   43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>   44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>   45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>   46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>   47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>   48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>   49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>   50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>   51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>   52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>   53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>   54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>   55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>   56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>   57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>   58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>   59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>   60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>   61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>   62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>   63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>   64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>   65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>   66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>   67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>   68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>   69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>   70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>   71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>   72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>   73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>   74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>   75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>   76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>   77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>   78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>   79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>   80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>   81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>   82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>   83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>   84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>   85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>   86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>   87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>   88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>   89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>   90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>   91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>   92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>   93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>   94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>   95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>   96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>   97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>   98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>   99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>  100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>  101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>  102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>  103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>  104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>  105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>  106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>  107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>  108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>  109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>  110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>  111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>  112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>  113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>  114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>  115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>  116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>  117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>  118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>  119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>  120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>  121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>  122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>  123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>  124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>  125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>  126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>  127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>  128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>  129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>  130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>  131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>  132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>  133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>  134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>  135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>  136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>  137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>  138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>  139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>  140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>  141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>  142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>  143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>  144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>  145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>  146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>  147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>  148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>  149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>  150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>  151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>  152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>  153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>  154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>  155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>  156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>  157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>  158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>  159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>  160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>  161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>  162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>  163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>  164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>  165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>  166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>  167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>  168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>  169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>  170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>  171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>  172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>  173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>  174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>  175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>  176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>  177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>  178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>  179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>  180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>  181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>  182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>  183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>  184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>  185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>  186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>  187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>  188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>  189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>  190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>  191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>  192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>  193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>  194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>  195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>  196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>  197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>  198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>  199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>  200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>  201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>  202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>  203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>  204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>  205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>  206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>  207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>  208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>  209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>  210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>> 
>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>> 
>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>> 
>>>>> Barry
>>>>> 
>>>>> 
>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>   0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>   2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>> 
>>>>> 
>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>> 
>>>>>>   0 SNES Function norm 1.132185384796e-08 
>>>>>>     0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>     1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>   1 SNES Function norm 2.177599365111e-12 
>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>   0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>   0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>   2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>   0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>   2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>   0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>   2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>   0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>   2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>   0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>   2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>   0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>   2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>   0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>   0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>   2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>   0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>   2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>   0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>   2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>   0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>   2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>   0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>   2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>   0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>   2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>   0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>   2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>   0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>   2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>   0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>   2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>   0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>   2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>   0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>   2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>   0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>   2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>   0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>   2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>   0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>   2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>   0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>   2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>   0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>   2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>   0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>   2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>   0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>   2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>   0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>   2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>   0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>   2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>   0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>   2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>   0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>   2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>   0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>   2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>   0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>   2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>   0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>   2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>   0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>   2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>   0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>   2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>   0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>   2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>   0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>   2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>   0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>   2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>   0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>   2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>   0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>   2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>   0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>   0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>   0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>   0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>   0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>   0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>   0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>   0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>> 
>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>> 
>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>> 
>>>>>>> This is a minor glitch we'll get past.
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>> 
>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hmm,
>>>>>>>>> 
>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>> 
>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>> 
>>>>>>>>> then 
>>>>>>>>> 
>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>> 
>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>> 
>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>> 
>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>   Mat Object:           1 MPI processes
>>>>>>>>>>>>>>     type: seqaij
>>>>>>>>>>>>>>     rows=15991, cols=15991
>>>>>>>>>>>>>>     package used to perform factorization: mumps
>>>>>>>>>>>>>>     total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>     total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>       MUMPS run parameters:
>>>>>>>>>>>>>>         SYM (matrix type):                   0 
>>>>>>>>>>>>>>         PAR (host participation):            1 
>>>>>>>>>>>>>>         ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>         ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>         ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>         ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>         ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>         ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>         ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>         ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>         ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>         ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>         ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>         ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>         ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>         ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>         ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>         ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>         ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>         ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>         ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>         ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>         ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>         ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>         ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>         ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>         ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>         ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>         ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>         ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>         CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>         CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>         CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>         CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>         CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>         RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>           [0] 1.95838e+06 
>>>>>>>>>>>>>>         RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>           [0]  143924 
>>>>>>>>>>>>>>         RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>           [0]  1.95943e+06 
>>>>>>>>>>>>>>         INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>         [0] 7 
>>>>>>>>>>>>>>         INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>           [0] 7 
>>>>>>>>>>>>>>         INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>           [0] 15991 
>>>>>>>>>>>>>>         RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>         RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>         RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>         (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>         INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>         INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>         INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>         INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>         INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>         INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>         INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>         INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>         INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>         INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>         INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>         INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>         INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>         INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>         INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>         INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>         INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>         INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>         INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>         INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>         INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>         INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>         INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>         INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>         INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>         INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>         INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>         INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>         INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>     SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>     SNESSetDM()
>>>>>>>>>>>>>>>>     etc
>>>>>>>>>>>>>>>>     SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>     SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>     SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>     VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>     SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/80d56e43/attachment-0001.html>

From knepley at gmail.com  Tue Sep  8 10:54:29 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Sep 2015 10:54:29 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
Message-ID: <CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>

On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Just to clarify, again, during the linear solve phase, even though the PC
> residual norm was small (getting to 10^{-10} in 2-3 iterations) that the
> true residual norm could be comparatively large signified there is
> something wrong.  The one thing i was curious about is why it seems that
> there is only an issue during grid sequencing.  I?m not seeing that kind of
> behavior on the coarsest mesh.
>

That definitely sounds like a bug in the Jacobian evaluation. Maybe you
assume a certain mesh size, and when
PETSc refines the DA and passes in the new one something goes wrong?

   Matt


> -gideon
>
> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Alright, I?ll take another look at what has been implemented in my
> Jacobian file.  One thing I?m a bit unclear about is do I need these
>  -snes_mf and snes_mf_operator flags if I have, coded in,
>
> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>
>
> Gideon,
>
>   In your case the DM creates that Jacobian matrix object and
> form_function_jacobian's job is just to fill it up with the correct values
> for the current solution value.   If you use -snes_mf_operator then SNES
> uses a matrix-free multiple for the first matrix and your Jacobian to build
> the preconditioner. With -snes_mf it uses a matrix-free multiple for the
> first and second matrix and ignores what you compute. Since it is matrix
> free in both it cannot build a preconditioner hence the -pc_type none
>
>  Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian
> you provide is correct
>
>
>  Barry
>
>
>
> -gideon
>
> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> If the Jacobian you provide is correct then using -snes_mf with a PC type
> of LU should give two iterations (or at most three) to get very good
> (10^-12) convergence of the linear system. Here you are not getting that
> after a few SNES iterations. So I am strongly leaning to something wrong
> with the Jacobian you provide.  -snes_type test unfortunately doesn't catch
> all problems.
>
> You can try -snes_mf -pc_type none and see how the linear solver
> converges.
>
>
> Barry
>
>
>
> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Off the top of my head, I can?t rule out a null space in this problem. I
> thought I did a pretty good job of coding the Jacobian by hand.  And before
> I made that change to the Jacobian file, changing J for Jpre, if I ran with
>  -snes_check_jacobian, it appeared to be satisfactory.
>
> When I run with -ksp_type fgmres, I get
>
>    0 SNES Function norm 3.857327250267e-09
>      0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm
> 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>      1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm
> 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>      2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm
> 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>    1 SNES Function norm 2.479697868412e-12
>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>  0 SNES Function norm 5.066222213176e+03
>    0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm
> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm
> 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>  1 SNES Function norm 8.484696862318e+02
>    0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm
> 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm
> 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>    2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm
> 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>  2 SNES Function norm 6.551139809777e+02
>    0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm
> 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm
> 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>    2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm
> 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>  3 SNES Function norm 5.771483337231e+02
>    0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm
> 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm
> 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>    2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm
> 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>  4 SNES Function norm 5.237540245670e+02
>    0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm
> 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm
> 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>    2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm
> 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>  5 SNES Function norm 4.753058321803e+02
>    0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm
> 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm
> 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>    2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm
> 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>  6 SNES Function norm 4.221482124411e+02
>    0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm
> 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm
> 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>    2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm
> 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>  7 SNES Function norm 3.803620028964e+02
>    0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm
> 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm
> 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>    2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm
> 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>  8 SNES Function norm 3.769519577570e+02
>    0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm
> 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm
> 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>    2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm
> 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>    3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm
> 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>  9 SNES Function norm 3.766389203406e+02
>    0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm
> 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm
> 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>    2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm
> 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
> 10 SNES Function norm 3.751822489648e+02
>    0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm
> 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm
> 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>    2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm
> 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
> 11 SNES Function norm 3.737189963925e+02
>    0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm
> 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm
> 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>    2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm
> 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>    3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm
> 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
> 12 SNES Function norm 3.737110331594e+02
>    0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm
> 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm
> 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>    2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm
> 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
> 13 SNES Function norm 3.684175513154e+02
>    0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm
> 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm
> 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>    2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm
> 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
> 14 SNES Function norm 3.509930783933e+02
>    0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm
> 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm
> 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>    2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm
> 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
> 15 SNES Function norm 3.162121096565e+02
>    0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm
> 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm
> 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>    2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm
> 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
> 16 SNES Function norm 2.890165052818e+02
>    0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm
> 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm
> 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>    2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm
> 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
> 17 SNES Function norm 2.816929421445e+02
>    0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm
> 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm
> 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>    2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm
> 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>    3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm
> 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
> 18 SNES Function norm 2.814407559738e+02
>    0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm
> 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm
> 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>    2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm
> 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
> 19 SNES Function norm 2.810261821835e+02
>    0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm
> 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm
> 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>    2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm
> 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
> 20 SNES Function norm 2.755723672077e+02
>    0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm
> 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm
> 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>    2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm
> 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>    3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm
> 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>    4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm
> 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>    5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm
> 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>    6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm
> 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
> 21 SNES Function norm 2.755720389981e+02
>    0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm
> 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm
> 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>    2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm
> 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>    3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm
> 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>    4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm
> 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>    5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm
> 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>    6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm
> 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
> 22 SNES Function norm 2.755716464881e+02
>    0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm
> 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm
> 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>    2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm
> 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>    3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm
> 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>    4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm
> 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
> 23 SNES Function norm 2.755713163168e+02
>    0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm
> 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm
> 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>    2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm
> 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>    3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm
> 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>    4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm
> 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>    5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm
> 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>    6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm
> 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>    7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm
> 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>    8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm
> 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
> 24 SNES Function norm 2.755712363532e+02
>    0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm
> 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm
> 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>    2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm
> 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>    3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm
> 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>    4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm
> 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>    5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm
> 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>    6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm
> 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>    7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm
> 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>    8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm
> 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>    9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm
> 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>   10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm
> 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>   11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm
> 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>   12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm
> 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>   13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm
> 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>   14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm
> 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>   15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm
> 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>   16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm
> 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>   17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm
> 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>   18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm
> 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>   19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm
> 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>   20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm
> 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>   21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm
> 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>   22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm
> 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>   23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm
> 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>   24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm
> 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>   25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm
> 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>   26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm
> 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>   27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm
> 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>   28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm
> 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>   29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm
> 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>   30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm
> 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>   31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm
> 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>   32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm
> 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>   33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm
> 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>   34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm
> 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>   35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm
> 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>   36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm
> 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>   37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm
> 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>   38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm
> 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>   39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm
> 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>   40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm
> 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>   41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm
> 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>   42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm
> 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>   43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm
> 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>   44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm
> 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>   45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm
> 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>   46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm
> 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>   47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm
> 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>   48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm
> 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>   49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm
> 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>   50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm
> 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>   51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm
> 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>   52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm
> 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>   53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm
> 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>   54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm
> 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>   55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm
> 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>   56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm
> 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>   57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm
> 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>   58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm
> 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>   59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm
> 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>   60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm
> 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>   61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm
> 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>   62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm
> 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>   63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm
> 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>   64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm
> 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>   65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm
> 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>   66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm
> 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>   67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm
> 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>   68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm
> 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>   69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm
> 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>   70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm
> 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>   71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm
> 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>   72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm
> 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>   73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm
> 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>   74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm
> 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>   75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm
> 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>   76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm
> 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>   77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm
> 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>   78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm
> 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>   79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm
> 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>   80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm
> 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>   81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm
> 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>   82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm
> 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>   83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm
> 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>   84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm
> 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>   85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm
> 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>   86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm
> 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>   87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm
> 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>   88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm
> 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>   89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm
> 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>   90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm
> 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>   91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm
> 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>   92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm
> 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>   93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm
> 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>   94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm
> 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>   95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm
> 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>   96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm
> 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>   97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm
> 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>   98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm
> 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>   99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm
> 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>  100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm
> 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>  101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm
> 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>  102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm
> 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>  103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm
> 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>  104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm
> 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>  105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm
> 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>  106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm
> 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>  107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm
> 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>  108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm
> 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>  109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm
> 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>  110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm
> 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>  111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm
> 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>  112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm
> 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>  113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm
> 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>  114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm
> 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>  115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm
> 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>  116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm
> 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>  117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm
> 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>  118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm
> 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>  119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm
> 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>  120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm
> 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>  121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm
> 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>  122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm
> 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>  123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm
> 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>  124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm
> 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>  125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm
> 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>  126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm
> 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>  127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm
> 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>  128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm
> 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>  129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm
> 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>  130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm
> 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>  131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm
> 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>  132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm
> 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>  133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm
> 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>  134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm
> 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>  135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm
> 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>  136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm
> 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>  137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm
> 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>  138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm
> 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>  139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm
> 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>  140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm
> 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>  141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm
> 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>  142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm
> 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>  143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm
> 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>  144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm
> 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>  145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm
> 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>  146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm
> 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>  147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm
> 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>  148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm
> 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>  149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm
> 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>  150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm
> 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>  151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm
> 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>  152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm
> 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>  153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm
> 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>  154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm
> 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>  155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm
> 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>  156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm
> 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>  157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm
> 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>  158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm
> 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>  159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm
> 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>  160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm
> 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>  161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm
> 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>  162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm
> 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>  163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm
> 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>  164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm
> 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>  165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm
> 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>  166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm
> 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>  167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm
> 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>  168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm
> 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>  169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm
> 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>  170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm
> 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>  171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm
> 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>  172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm
> 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>  173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm
> 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>  174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm
> 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>  175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm
> 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>  176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm
> 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>  177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm
> 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>  178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm
> 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>  179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm
> 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>  180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm
> 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>  181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm
> 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>  182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm
> 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>  183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm
> 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>  184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm
> 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>  185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm
> 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>  186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm
> 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>  187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm
> 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>  188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm
> 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>  189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm
> 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>  190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm
> 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>  191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm
> 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>  192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm
> 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>  193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm
> 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>  194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm
> 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>  195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm
> 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>  196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm
> 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>  197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm
> 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>  198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm
> 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>  199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm
> 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>  200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm
> 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>  201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm
> 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>  202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm
> 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>  203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm
> 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>  204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm
> 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>  205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm
> 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>  206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm
> 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>  207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm
> 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>  208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm
> 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>  209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm
> 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>  210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm
> 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>
> -gideon
>
> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Is there any chance your Jacobian may have a null space? (For example with
> Neumann boundary conditions there is often a nullspace)
>
> If not run with with -ksp_type fgmres and send the same output.
>
> Something is screwy with the linear system convergence, could be several
> things including either a null space problem or that your Jacobian that you
> provide is terribly wrong.
>
> Barry
>
>
> 21 SNES Function norm 2.987064584431e+02
>   0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm
> 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm
> 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>   2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm
> 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>
>
> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Got it, fixing that, and returning to the original question, this is what
> I now get, when I use those two flags:
>
>   0 SNES Function norm 1.132185384796e-08
>     0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm
> 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm
> 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>   1 SNES Function norm 2.177599365111e-12
> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
> 0 SNES Function norm 5.066222213176e+03
>   0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm
> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm
> 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
> 1 SNES Function norm 8.482593852817e+02
>   0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm
> 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm
> 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>   2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm
> 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
> 2 SNES Function norm 6.543140468549e+02
>   0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm
> 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm
> 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>   2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm
> 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
> 3 SNES Function norm 5.766430557220e+02
>   0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm
> 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm
> 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>   2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm
> 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
> 4 SNES Function norm 5.235211958260e+02
>   0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm
> 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm
> 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>   2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm
> 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
> 5 SNES Function norm 4.752913229649e+02
>   0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm
> 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm
> 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>   2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm
> 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
> 6 SNES Function norm 4.220255380391e+02
>   0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm
> 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm
> 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>   2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm
> 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
> 7 SNES Function norm 3.805408907074e+02
>   0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm
> 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm
> 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
> 8 SNES Function norm 3.764619752339e+02
>   0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm
> 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm
> 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>   2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm
> 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
> 9 SNES Function norm 3.761182227091e+02
>   0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm
> 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm
> 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>   2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm
> 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
> 10 SNES Function norm 3.740017190063e+02
>   0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm
> 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm
> 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>   2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm
> 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
> 11 SNES Function norm 3.725903477238e+02
>   0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm
> 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm
> 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>   2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm
> 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
> 12 SNES Function norm 3.716162097231e+02
>   0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm
> 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm
> 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>   2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm
> 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
> 13 SNES Function norm 3.674168632847e+02
>   0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm
> 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm
> 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>   2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm
> 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
> 14 SNES Function norm 3.532395445266e+02
>   0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm
> 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm
> 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>   2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm
> 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
> 15 SNES Function norm 3.182438872366e+02
>   0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm
> 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm
> 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>   2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm
> 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
> 16 SNES Function norm 3.091759892779e+02
>   0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm
> 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm
> 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>   2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm
> 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
> 17 SNES Function norm 2.987839504359e+02
>   0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm
> 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm
> 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>   2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm
> 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
> 18 SNES Function norm 2.987073622777e+02
>   0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm
> 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm
> 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>   2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm
> 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
> 19 SNES Function norm 2.987067936734e+02
>   0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm
> 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm
> 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>   2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm
> 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
> 20 SNES Function norm 2.987067502910e+02
>   0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm
> 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm
> 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>   2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm
> 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
> 21 SNES Function norm 2.987064584431e+02
>   0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm
> 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm
> 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>   2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm
> 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
> 22 SNES Function norm 2.987064525262e+02
>   0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm
> 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm
> 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>   2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm
> 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
> 23 SNES Function norm 2.987064121622e+02
>   0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm
> 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm
> 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>   2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm
> 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
> 24 SNES Function norm 2.987063973426e+02
>   0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm
> 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm
> 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>   2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm
> 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
> 25 SNES Function norm 2.987063920553e+02
>   0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm
> 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm
> 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>   2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm
> 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
> 26 SNES Function norm 2.987063919786e+02
>   0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm
> 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm
> 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>   2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm
> 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
> 27 SNES Function norm 2.987063393145e+02
>   0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm
> 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm
> 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>   2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm
> 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
> 28 SNES Function norm 2.987063253277e+02
>   0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm
> 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm
> 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>   2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm
> 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
> 29 SNES Function norm 2.987063197153e+02
>   0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm
> 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm
> 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>   2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm
> 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
> 30 SNES Function norm 2.987063193089e+02
>   0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm
> 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm
> 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>   2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm
> 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
> 31 SNES Function norm 2.987063192061e+02
>   0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm
> 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm
> 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>   2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm
> 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
> 32 SNES Function norm 2.987063094492e+02
>   0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm
> 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm
> 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>   2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm
> 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
> 33 SNES Function norm 2.987063089942e+02
>   0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm
> 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm
> 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>   2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm
> 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
> 34 SNES Function norm 2.987062660109e+02
>   0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm
> 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm
> 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>   2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm
> 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
> 35 SNES Function norm 2.987062107900e+02
>   0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm
> 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm
> 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>   2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm
> 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
> 36 SNES Function norm 2.987062055224e+02
>   0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm
> 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm
> 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>   2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm
> 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
> 37 SNES Function norm 2.987061774798e+02
>   0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm
> 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm
> 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>   2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm
> 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
> 38 SNES Function norm 2.987061715400e+02
>   0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm
> 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm
> 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>   2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm
> 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
> 39 SNES Function norm 2.987061699634e+02
>   0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm
> 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm
> 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>   2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm
> 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
> 40 SNES Function norm 2.987061630064e+02
>   0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm
> 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
> 41 SNES Function norm 2.987061630064e+02
>   0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm
> 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
> 42 SNES Function norm 2.987061630064e+02
>   0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm
> 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
> 43 SNES Function norm 2.987061630064e+02
>   0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm
> 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
> 44 SNES Function norm 2.987061630064e+02
>   0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm
> 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
> 45 SNES Function norm 2.987061630064e+02
>   0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm
> 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
> 46 SNES Function norm 2.987061630064e+02
>   0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm
> 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
> 47 SNES Function norm 2.987061630064e+02
>   0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm
> 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>
>
> -gideon
>
> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> This indicates that somewhere in your ComputeJacobian you are setting
> matrix entries with the first Mat argument when you should always set them
> with the second matrix argument. For example if you have
>
> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>
> you should call all the MatSetValues() with jpre, no J. Then at the end of
> the function you should call MatAssemblyBegin/End() on jpre then on J if J
> is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>
> This is a minor glitch we'll get past.
>
> Barry
>
> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> By the way, I tried using a different petsc installation, and now, rather
> than the segmentation fault, I get the following error:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Mat type mffd
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named
> gs_air by gideon Mon Sep  7 21:32:18 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes
> --download-suitesparse=yes --download-superlu=yes
> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes
> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes
> --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in
> /opt/petsc-3.5.4/src/mat/interface/matrix.c
>
> -gideon
>
> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Hmm,
>
> Ok you can try running it directly in the debugger since it is one
> process, type
>
> gdb ./blowup_batch_refine
>
> then
>
> when the debugger comes up (if it does not cut and paste all output and
> send it)
>
> run -on_error_abort -snes_mf_operator  and any other options you normally
> use
>
>
> Barry
>
> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Running with that flag gives me this:
>
> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111
> on gs_air
> Unable to start debugger: No such file or directory
>
>
>
> -gideon
>
> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> This should not happen. Run with a debug version of PETSc installed and
> the option -start_in_debugger noxterm  Once the debugger starts up type
> cont and when it crashes type where or bt  Send all output
>
>
>
> Barry
>
>
> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> I?m getting an error with -snes_mf_operator,
>
> 0 SNES Function norm 1.421454390131e-02
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by
> gideon Mon Sep  7 21:08:19 2015
> [0]PETSC ERROR: Configure options --prefix=/opt/local
> --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries
> --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0
> --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate
> --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local
> --with-superlu-dir=/opt/local --with-metis-dir=/opt/local
> --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local
> --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local
> CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp
> FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp
> F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os
> FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names"
> CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os
> FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports
> --with-mpiexec=mpiexec-mpich-mp
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>
> -gideon
>
> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc
> SNES is generating a poor descent direction. You can try
> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What
> happens?
>
> Barry
>
>
>
> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> No problem Matt, I don?t think we had previously discussed that output.
> Here is a case where things fail.
>
> 0 SNES Function norm 4.027481756921e-09
> 1 SNES Function norm 1.760477878365e-12
> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
> 0 SNES Function norm 5.066222213176e+03
> 1 SNES Function norm 8.484697184230e+02
> 2 SNES Function norm 6.549559723294e+02
> 3 SNES Function norm 5.770723278153e+02
> 4 SNES Function norm 5.237702240594e+02
> 5 SNES Function norm 4.753909019848e+02
> 6 SNES Function norm 4.221784590755e+02
> 7 SNES Function norm 3.806525080483e+02
> 8 SNES Function norm 3.762054656019e+02
> 9 SNES Function norm 3.758975226873e+02
> 10 SNES Function norm 3.757032042706e+02
> 11 SNES Function norm 3.728798164234e+02
> 12 SNES Function norm 3.723078741075e+02
> 13 SNES Function norm 3.721848059825e+02
> 14 SNES Function norm 3.720227575629e+02
> 15 SNES Function norm 3.720051998555e+02
> 16 SNES Function norm 3.718945430587e+02
> 17 SNES Function norm 3.700412694044e+02
> 18 SNES Function norm 3.351964889461e+02
> 19 SNES Function norm 3.096016086233e+02
> 20 SNES Function norm 3.008410789787e+02
> 21 SNES Function norm 2.752316716557e+02
> 22 SNES Function norm 2.707658474165e+02
> 23 SNES Function norm 2.698436736049e+02
> 24 SNES Function norm 2.618233857172e+02
> 25 SNES Function norm 2.600121920634e+02
> 26 SNES Function norm 2.585046423168e+02
> 27 SNES Function norm 2.568551090220e+02
> 28 SNES Function norm 2.556404537064e+02
> 29 SNES Function norm 2.536353523683e+02
> 30 SNES Function norm 2.533596070171e+02
> 31 SNES Function norm 2.532324379596e+02
> 32 SNES Function norm 2.531842335211e+02
> 33 SNES Function norm 2.531684527520e+02
> 34 SNES Function norm 2.531637604618e+02
> 35 SNES Function norm 2.531624767821e+02
> 36 SNES Function norm 2.531621359093e+02
> 37 SNES Function norm 2.531620504925e+02
> 38 SNES Function norm 2.531620350055e+02
> 39 SNES Function norm 2.531620310522e+02
> 40 SNES Function norm 2.531620300471e+02
> 41 SNES Function norm 2.531620298084e+02
> 42 SNES Function norm 2.531620297478e+02
> 43 SNES Function norm 2.531620297324e+02
> 44 SNES Function norm 2.531620297303e+02
> 45 SNES Function norm 2.531620297302e+02
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
> 0 SNES Function norm 9.636339304380e+03
> 1 SNES Function norm 8.997731184634e+03
> 2 SNES Function norm 8.120498349232e+03
> 3 SNES Function norm 7.322379894820e+03
> 4 SNES Function norm 6.599581599149e+03
> 5 SNES Function norm 6.374872854688e+03
> 6 SNES Function norm 6.372518007653e+03
> 7 SNES Function norm 6.073996314301e+03
> 8 SNES Function norm 5.635965277054e+03
> 9 SNES Function norm 5.155389064046e+03
> 10 SNES Function norm 5.080567902638e+03
> 11 SNES Function norm 5.058878643969e+03
> 12 SNES Function norm 5.058835649793e+03
> 13 SNES Function norm 5.058491285707e+03
> 14 SNES Function norm 5.057452865337e+03
> 15 SNES Function norm 5.057226140688e+03
> 16 SNES Function norm 5.056651272898e+03
> 17 SNES Function norm 5.056575190057e+03
> 18 SNES Function norm 5.056574632598e+03
> 19 SNES Function norm 5.056574520229e+03
> 20 SNES Function norm 5.056574492569e+03
> 21 SNES Function norm 5.056574485124e+03
> 22 SNES Function norm 5.056574483029e+03
> 23 SNES Function norm 5.056574482427e+03
> 24 SNES Function norm 5.056574482302e+03
> 25 SNES Function norm 5.056574482287e+03
> 26 SNES Function norm 5.056574482282e+03
> 27 SNES Function norm 5.056574482281e+03
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
> SNES Object: 1 MPI processes
> type: newtonls
> maximum iterations=50, maximum function evaluations=10000
> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
> total number of linear solver iterations=28
> total number of function evaluations=323
> total number of grid sequence refinements=2
> SNESLineSearch Object:   1 MPI processes
> type: bt
> interpolation: cubic
> alpha=1.000000e-04
> maxstep=1.000000e+08, minlambda=1.000000e-12
> tolerances: relative=1.000000e-08, absolute=1.000000e-15,
> lambda=1.000000e-08
> maximum iterations=40
> KSP Object:   1 MPI processes
> type: gmres
> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=10000, initial guess is zero
> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using PRECONDITIONED norm type for convergence test
> PC Object:   1 MPI processes
> type: lu
> LU: out-of-place factorization
> tolerance for zero pivot 2.22045e-14
> matrix ordering: nd
> factor fill ratio given 0, needed 0
> Factored matrix follows:
>   Mat Object:           1 MPI processes
>     type: seqaij
>     rows=15991, cols=15991
>     package used to perform factorization: mumps
>     total: nonzeros=255801, allocated nonzeros=255801
>     total number of mallocs used during MatSetValues calls =0
>       MUMPS run parameters:
>         SYM (matrix type):                   0
>         PAR (host participation):            1
>         ICNTL(1) (output for error):         6
>         ICNTL(2) (output of diagnostic msg): 0
>         ICNTL(3) (output for global info):   0
>         ICNTL(4) (level of printing):        0
>         ICNTL(5) (input mat struct):         0
>         ICNTL(6) (matrix prescaling):        7
>         ICNTL(7) (sequentia matrix ordering):6
>         ICNTL(8) (scalling strategy):        77
>         ICNTL(10) (max num of refinements):  0
>         ICNTL(11) (error analysis):          0
>         ICNTL(12) (efficiency control):                         1
>         ICNTL(13) (efficiency control):                         0
>         ICNTL(14) (percentage of estimated workspace increase): 20
>         ICNTL(18) (input mat struct):                           0
>         ICNTL(19) (Shur complement info):                       0
>         ICNTL(20) (rhs sparse pattern):                         0
>         ICNTL(21) (somumpstion struct):                            0
>         ICNTL(22) (in-core/out-of-core facility):               0
>         ICNTL(23) (max size of memory can be allocated locally):0
>         ICNTL(24) (detection of null pivot rows):               0
>         ICNTL(25) (computation of a null space basis):          0
>         ICNTL(26) (Schur options for rhs or solution):          0
>         ICNTL(27) (experimental parameter):                     -8
>         ICNTL(28) (use parallel or sequential ordering):        1
>         ICNTL(29) (parallel ordering):                          0
>         ICNTL(30) (user-specified set of entries in inv(A)):    0
>         ICNTL(31) (factors is discarded in the solve phase):    0
>         ICNTL(33) (compute determinant):                        0
>         CNTL(1) (relative pivoting threshold):      0.01
>         CNTL(2) (stopping criterion of refinement): 1.49012e-08
>         CNTL(3) (absomumpste pivoting threshold):      0
>         CNTL(4) (vamumpse of static pivoting):         -1
>         CNTL(5) (fixation for null pivots):         0
>         RINFO(1) (local estimated flops for the elimination after
> analysis):
>           [0] 1.95838e+06
>         RINFO(2) (local estimated flops for the assembly after
> factorization):
>           [0]  143924
>         RINFO(3) (local estimated flops for the elimination after
> factorization):
>           [0]  1.95943e+06
>         INFO(15) (estimated size of (in MB) MUMPS internal data for
> running numerical factorization):
>         [0] 7
>         INFO(16) (size of (in MB) MUMPS internal data used during
> numerical factorization):
>           [0] 7
>         INFO(23) (num of pivots eliminated on this processor after
> factorization):
>           [0] 15991
>         RINFOG(1) (global estimated flops for the elimination after
> analysis): 1.95838e+06
>         RINFOG(2) (global estimated flops for the assembly after
> factorization): 143924
>         RINFOG(3) (global estimated flops for the elimination after
> factorization): 1.95943e+06
>         (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>         INFOG(3) (estimated real workspace for factors on all processors
> after analysis): 255801
>         INFOG(4) (estimated integer workspace for factors on all
> processors after analysis): 127874
>         INFOG(5) (estimated maximum front size in the complete tree): 11
>         INFOG(6) (number of nodes in the complete tree): 3996
>         INFOG(7) (ordering option effectively use after analysis): 6
>         INFOG(8) (structural symmetry in percent of the permuted matrix
> after analysis): 86
>         INFOG(9) (total real/complex workspace to store the matrix factors
> after factorization): 255865
>         INFOG(10) (total integer space store the matrix factors after
> factorization): 127890
>         INFOG(11) (order of largest frontal matrix after factorization):
> 11
>         INFOG(12) (number of off-diagonal pivots): 19
>         INFOG(13) (number of delayed pivots after factorization): 8
>         INFOG(14) (number of memory compress after factorization): 0
>         INFOG(15) (number of steps of iterative refinement after
> solution): 0
>         INFOG(16) (estimated size (in MB) of all MUMPS internal data for
> factorization after analysis: value on the most memory consuming
> processor): 7
>         INFOG(17) (estimated size of all MUMPS internal data for
> factorization after analysis: sum over all processors): 7
>         INFOG(18) (size of all MUMPS internal data allocated during
> factorization: value on the most memory consuming processor): 7
>         INFOG(19) (size of all MUMPS internal data allocated during
> factorization: sum over all processors): 7
>         INFOG(20) (estimated number of entries in the factors): 255801
>         INFOG(21) (size in MB of memory effectively used during
> factorization - value on the most memory consuming processor): 7
>         INFOG(22) (size in MB of memory effectively used during
> factorization - sum over all processors): 7
>         INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0
>         INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1
>         INFOG(25) (after factorization: number of pivots modified by
> static pivoting): 0
>         INFOG(28) (after factorization: number of null pivots
> encountered): 0
>         INFOG(29) (after factorization: effective number of entries in the
> factors (sum over all processors)): 255865
>         INFOG(30, 31) (after solution: size in Mbytes of memory used
> during solution phase): 5, 5
>         INFOG(32) (after analysis: type of analysis done): 1
>         INFOG(33) (value used for ICNTL(8)): 7
>         INFOG(34) (exponent of the determinant if determinant is
> requested): 0
> linear system matrix = precond matrix:
> Mat Object:     1 MPI processes
> type: seqaij
> rows=15991, cols=15991
> total: nonzeros=223820, allocated nonzeros=431698
> total number of mallocs used during MatSetValues calls =15991
> using I-node routines: found 4000 nodes, limit used is 5
>
>
>
>
> -gideon
>
> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> Barry,
>
> I finally got a chance to really try using the grid sequencing within my
> code.  I find that, in some cases, even if it can solve successfully on the
> coarsest mesh, the SNES fails, usually due to a line search failure, when
> it tries to compute along the grid sequence.  Would you have any
> suggestions?
>
> I apologize if I have asked before, but can you give me -snes_view for the
> solver? I could not find it in the email thread.
>
> I would suggest trying to fiddle with the line search, or precondition it
> with Richardson. It would be nice to see -snes_monitor
> for the runs that fail, and then we can break down the residual into
> fields and look at it again (if my custom residual monitor
> does not work we can write one easily). Seeing which part of the residual
> does not converge is key to designing the NASM
> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai,
> present it. We need better monitoring in PETSc.
>
> Thanks,
>
> Matt
>
> -gideon
>
> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Yes, if i continue in this parameter on the coarse mesh, I can generally
> solve at all values. I do find that I need to do some amount of
> continuation to solve near the endpoint.  The problem is that on the coarse
> mesh, things are not fully resolved at all the values along the
> continuation parameter, and I would like to do refinement.
>
> One subtlety is that I actually want the intermediate continuation
> solutions  too.  Currently, without doing any grid sequence, I compute
> each, write it to disk, and then go on to the next one.  So I now need to
> go back an refine them.  I was thinking that perhaps I could refine them on
> the fly, dump them to disk, and use the coarse solution as the starting
> guess at the next iteration, but that would seem to require resetting the
> snes back to the coarse grid.
>
> The alternative would be to just script the mesh refinement in a post
> processing stage, where each value of the continuation is parameter is
> loaded on the coarse mesh, and refined.  Perhaps that?s the most practical
> thing to do.
>
>
> I would do the following. Create your DM and create a SNES that will do
> the continuation
>
> loop over continuation parameter
>
> SNESSolve(snes,NULL,Ucoarse);
>
> if (you decide you want to see the refined solution at this continuation
> point) {
>     SNESCreate(comm,&snesrefine);
>     SNESSetDM()
>     etc
>     SNESSetGridSequence(snesrefine,)
>     SNESSolve(snesrefine,0,Ucoarse);
>     SNESGetSolution(snesrefine,&Ufine);
>     VecView(Ufine or do whatever you want to do with the Ufine at that
> continuation point
>     SNESDestroy(snesrefine);
> end if
>
> end loop over continuation parameter.
>
> Barry
>
>
> -gideon
>
> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>
> 3.  This problem is actually part of a continuation problem that roughly
> looks like this
>
> for( continuation parameter p = 0 to 1){
>
> solve with parameter p_i using solution from p_{i-1},
> }
>
> What I would like to do is to start the solver, for each value of
> parameter p_i on the coarse mesh, and then do grid sequencing on that.  But
> it appears that after doing grid sequencing on the initial p_0 = 0, the
> SNES is set to use the finer mesh.
>
>
> So you are using continuation to give you a good enough initial guess on
> the coarse level to even get convergence on the coarse level? First I would
> check if you even need the continuation (or can you not even solve the
> coarse problem without it).
>
> If you do need the continuation then you will need to tweak how you do the
> grid sequencing. I think this will work:
>
> Do not use -snes_grid_sequencing
>
> Run SNESSolve() as many times as you want with your continuation
> parameter. This will all happen on the coarse mesh.
>
> Call SNESSetGridSequence()
>
> Then call SNESSolve() again and it will do one solve on the coarse level
> and then interpolate to the next level etc.
>
>
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/e4fc7e3f/attachment-0001.html>

From gideon.simpson at gmail.com  Tue Sep  8 14:20:37 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 15:20:37 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
Message-ID: <5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>

I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?


-gideon

> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
> 
> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
> PETSc refines the DA and passes in the new one something goes wrong?
> 
>    Matt
>  
> -gideon
> 
>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>> 
>> 
>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>> 
>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>> 
>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>> 
>> Gideon,
>> 
>>   In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>> 
>>  Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>> 
>> 
>>  Barry
>> 
>>> 
>>> 
>>> -gideon
>>> 
>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>> 
>>>> 
>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>> 
>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>> 
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>> 
>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>> 
>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>> 
>>>>> When I run with -ksp_type fgmres, I get
>>>>> 
>>>>>    0 SNES Function norm 3.857327250267e-09 
>>>>>      0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>      1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>      2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>    1 SNES Function norm 2.479697868412e-12 
>>>>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>  0 SNES Function norm 5.066222213176e+03 
>>>>>    0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>  1 SNES Function norm 8.484696862318e+02 
>>>>>    0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>    2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>  2 SNES Function norm 6.551139809777e+02 
>>>>>    0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>    2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>  3 SNES Function norm 5.771483337231e+02 
>>>>>    0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>    2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>  4 SNES Function norm 5.237540245670e+02 
>>>>>    0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>    2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>  5 SNES Function norm 4.753058321803e+02 
>>>>>    0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>    2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>  6 SNES Function norm 4.221482124411e+02 
>>>>>    0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>    2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>  7 SNES Function norm 3.803620028964e+02 
>>>>>    0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>    2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>  8 SNES Function norm 3.769519577570e+02 
>>>>>    0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>    2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>    3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>  9 SNES Function norm 3.766389203406e+02 
>>>>>    0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>    2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>    0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>    2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>    0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>    2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>    3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>    0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>    2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>    0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>    2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>    0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>    2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>    0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>    2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>    0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>    2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>    0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>    2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>    3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>    0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>    2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>    0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>    2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>    0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>    2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>    3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>    4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>    5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>    6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>    0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>    2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>    3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>    4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>    5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>    6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>    0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>    2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>    3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>    4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>    0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>    2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>    3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>    4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>    5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>    6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>    7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>    8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>    0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>    1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>    2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>    3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>    4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>    5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>    6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>    7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>    8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>    9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>   10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>   11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>   12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>   13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>   14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>   15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>   16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>   17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>   18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>   19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>   20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>   21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>   22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>   23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>   24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>   25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>   26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>   27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>   28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>   29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>   30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>   31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>   32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>   33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>   34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>   35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>   36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>   37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>   38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>   39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>   40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>   41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>   42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>   43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>   44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>   45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>   46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>   47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>   48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>   49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>   50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>   51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>   52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>   53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>   54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>   55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>   56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>   57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>   58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>   59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>   60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>   61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>   62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>   63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>   64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>   65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>   66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>   67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>   68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>   69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>   70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>   71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>   72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>   73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>   74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>   75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>   76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>   77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>   78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>   79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>   80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>   81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>   82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>   83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>   84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>   85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>   86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>   87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>   88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>   89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>   90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>   91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>   92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>   93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>   94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>   95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>   96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>   97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>   98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>   99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>  100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>  101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>  102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>  103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>  104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>  105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>  106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>  107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>  108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>  109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>  110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>  111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>  112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>  113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>  114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>  115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>  116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>  117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>  118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>  119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>  120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>  121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>  122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>  123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>  124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>  125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>  126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>  127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>  128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>  129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>  130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>  131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>  132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>  133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>  134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>  135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>  136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>  137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>  138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>  139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>  140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>  141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>  142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>  143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>  144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>  145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>  146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>  147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>  148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>  149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>  150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>  151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>  152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>  153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>  154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>  155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>  156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>  157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>  158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>  159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>  160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>  161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>  162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>  163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>  164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>  165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>  166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>  167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>  168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>  169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>  170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>  171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>  172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>  173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>  174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>  175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>  176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>  177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>  178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>  179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>  180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>  181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>  182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>  183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>  184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>  185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>  186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>  187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>  188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>  189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>  190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>  191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>  192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>  193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>  194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>  195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>  196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>  197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>  198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>  199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>  200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>  201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>  202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>  203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>  204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>  205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>  206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>  207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>  208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>  209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>  210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>> 
>>>>>> 
>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>> 
>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>> 
>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> 
>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>   0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>   2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>> 
>>>>>> 
>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>> 
>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>> 
>>>>>>>   0 SNES Function norm 1.132185384796e-08 
>>>>>>>     0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>     1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>   1 SNES Function norm 2.177599365111e-12 
>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>   0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>   0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>   2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>   2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>   2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>   2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>   2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>   0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>   2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>   0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>   0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>   2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>   0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>   2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>   2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>   2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>   0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>   2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>   0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>   2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>   0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>   2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>   2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>   0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>   2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>   0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>   2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>   0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>   2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>   0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>   2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>   0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>   2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>   0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>   2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>   2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>   2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>   0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>   2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>   2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>   2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>   0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>   2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>   0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>   2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>   2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>   0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>   2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>   0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>   2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>   2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>   2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>   0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>   2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>   0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>   2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>   0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>   2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>   0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>   2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>   0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>   2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>   0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>   2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>   0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>   0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>   0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>   0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>   0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>   0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>   0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>   0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>> 
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>> 
>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>> 
>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>> 
>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>> 
>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>> 
>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Hmm,
>>>>>>>>>> 
>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>> 
>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>> 
>>>>>>>>>> then 
>>>>>>>>>> 
>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>> 
>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>> 
>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>   Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>     type: seqaij
>>>>>>>>>>>>>>>     rows=15991, cols=15991
>>>>>>>>>>>>>>>     package used to perform factorization: mumps
>>>>>>>>>>>>>>>     total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>     total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>       MUMPS run parameters:
>>>>>>>>>>>>>>>         SYM (matrix type):                   0 
>>>>>>>>>>>>>>>         PAR (host participation):            1 
>>>>>>>>>>>>>>>         ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>         ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>         ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>         ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>         ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>         ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>         ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>         ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>         ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>         ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>         ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>         ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>         ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>         ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>         ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>         ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>         ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>         ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>         ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>         ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>         ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>         ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>         ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>         ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>         ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>         ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>         ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>         ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>         CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>         CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>         CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>         CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>         CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>         RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>           [0] 1.95838e+06 
>>>>>>>>>>>>>>>         RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>           [0]  143924 
>>>>>>>>>>>>>>>         RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>           [0]  1.95943e+06 
>>>>>>>>>>>>>>>         INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>         [0] 7 
>>>>>>>>>>>>>>>         INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>           [0] 7 
>>>>>>>>>>>>>>>         INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>           [0] 15991 
>>>>>>>>>>>>>>>         RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>         RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>         RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>         (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>         INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>         INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>         INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>         INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>         INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>         INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>         INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>         INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>         INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>         INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>         INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>         INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>         INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>         INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>         INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>         INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>         INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>         INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>         INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>         INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>         INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>         INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>         INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>         INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>         INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>         INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>         INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>         INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>         INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>     SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>     SNESSetDM()
>>>>>>>>>>>>>>>>>     etc
>>>>>>>>>>>>>>>>>     SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>     SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>     SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>     VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>     SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/5900c464/attachment-0001.html>

From rachitp at vt.edu  Tue Sep  8 16:47:57 2015
From: rachitp at vt.edu (Rachit Prasad)
Date: Tue, 8 Sep 2015 17:47:57 -0400
Subject: [petsc-users] Query regarding convergence of KSP solver
Message-ID: <CADe0GokDkBbzV1Mp8oZXk6SYyHTnkVjPCk+F6-Ny8adJXhCcqg@mail.gmail.com>

Hello,

I am working to solve a linear system of equations using a KSP solver. The
size of the matrix is approximately ~370,000. The following settings have
been used to set up the solver"

[image: Inline image 1]

The L2 norm of the RHS for this problem is ~365, which has been calculated
using the subroutine *VecNorm*. When I solve this system, I am finding that
irrespective of the number of maximum iterations the L2 norm of final
residual is pretty much constant around ~37860. The L2 norm of the residual
has been found using the subroutine *KSPGetResidualNorm. *The following is
how the residual is varying for different maximum iterations.
*[image: Inline image 2]*

With such being the case, I have the following doubts to clarify:

   - Are the settings used by me for the KSP solver adequate enough? Or am
   I missing something?
   - What could be reasons for my KSP solver to not converge below the
   permissible tolerance?
   - For a linear system of given size, how can one determine the number of
   maximum iterations that should be set for the KSP solver? Is there a thumb
   rule?


Regards,
Rachit Prasad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/fc6618ce/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: KSP residual convergence.JPG
Type: image/jpeg
Size: 30079 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/fc6618ce/attachment-0002.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: KSP settings.JPG
Type: image/jpeg
Size: 34260 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/fc6618ce/attachment-0003.jpe>

From jed at jedbrown.org  Tue Sep  8 16:55:45 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 08 Sep 2015 15:55:45 -0600
Subject: [petsc-users] Query regarding convergence of KSP solver
In-Reply-To: <CADe0GokDkBbzV1Mp8oZXk6SYyHTnkVjPCk+F6-Ny8adJXhCcqg@mail.gmail.com>
References: <CADe0GokDkBbzV1Mp8oZXk6SYyHTnkVjPCk+F6-Ny8adJXhCcqg@mail.gmail.com>
Message-ID: <87oahc7ezi.fsf@jedbrown.org>

Rachit Prasad <rachitp at vt.edu> writes:

> Hello,
>
> I am working to solve a linear system of equations using a KSP solver. The
> size of the matrix is approximately ~370,000. The following settings have
> been used to set up the solver"
>
> [image: Inline image 1]
>
> The L2 norm of the RHS for this problem is ~365, which has been calculated
> using the subroutine *VecNorm*. When I solve this system, I am finding that
> irrespective of the number of maximum iterations the L2 norm of final
> residual is pretty much constant around ~37860. 

Your solver has stagnated, either due to numerical precision issues
(e.g., if the matrix entries are enormous in this case) or (more likely)
due to an inadequate/unstable preconditioner.

> The L2 norm of the residual has been found using the subroutine
> *KSPGetResidualNorm. *The following is how the residual is varying for
> different maximum iterations.  *[image: Inline image 2]*
>
> With such being the case, I have the following doubts to clarify:
>
>    - Are the settings used by me for the KSP solver adequate enough? Or am
>    I missing something?
>    - What could be reasons for my KSP solver to not converge below the
>    permissible tolerance?
>    - For a linear system of given size, how can one determine the number of
>    maximum iterations that should be set for the KSP solver? Is there a thumb
>    rule?
>
>
> Regards,
> Rachit Prasad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/99ef3f2a/attachment.pgp>

From bsmith at mcs.anl.gov  Tue Sep  8 18:18:46 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 Sep 2015 18:18:46 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
Message-ID: <81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>


   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output

  Barry

> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
> 
> 
> -gideon
> 
>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>> 
>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>> PETSc refines the DA and passes in the new one something goes wrong?
>> 
>>    Matt
>>  
>> -gideon
>> 
>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>> 
>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>> 
>>> Gideon,
>>> 
>>>   In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>> 
>>>  Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>> 
>>> 
>>>  Barry
>>> 
>>>> 
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>> 
>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>> 
>>>>> 
>>>>> Barry
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>> 
>>>>>> When I run with -ksp_type fgmres, I get
>>>>>> 
>>>>>>    0 SNES Function norm 3.857327250267e-09 
>>>>>>      0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>      1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>      2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>    1 SNES Function norm 2.479697868412e-12 
>>>>>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>  0 SNES Function norm 5.066222213176e+03 
>>>>>>    0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>  1 SNES Function norm 8.484696862318e+02 
>>>>>>    0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>    2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>  2 SNES Function norm 6.551139809777e+02 
>>>>>>    0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>    2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>  3 SNES Function norm 5.771483337231e+02 
>>>>>>    0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>    2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>  4 SNES Function norm 5.237540245670e+02 
>>>>>>    0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>    2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>  5 SNES Function norm 4.753058321803e+02 
>>>>>>    0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>    2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>  6 SNES Function norm 4.221482124411e+02 
>>>>>>    0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>    2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>  7 SNES Function norm 3.803620028964e+02 
>>>>>>    0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>    2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>  8 SNES Function norm 3.769519577570e+02 
>>>>>>    0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>    2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>    3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>  9 SNES Function norm 3.766389203406e+02 
>>>>>>    0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>    2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>    0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>    2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>    0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>    2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>    3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>    0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>    2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>    0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>    2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>    0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>    2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>    0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>    2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>    0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>    2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>    0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>    2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>    3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>    0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>    2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>    0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>    2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>    0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>    2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>    3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>    4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>    5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>    6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>    0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>    2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>    3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>    4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>    5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>    6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>    0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>    2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>    3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>    4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>    0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>    2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>    3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>    4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>    5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>    6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>    7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>    8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>    0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>    2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>    3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>    4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>    5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>    6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>    7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>    8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>    9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>   10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>   11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>   12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>   13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>   14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>   15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>   16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>   17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>   18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>   19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>   20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>   21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>   22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>   23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>   24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>   25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>   26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>   27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>   28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>   29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>   30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>   31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>   32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>   33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>   34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>   35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>   36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>   37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>   38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>   39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>   40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>   41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>   42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>   43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>   44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>   45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>   46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>   47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>   48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>   49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>   50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>   51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>   52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>   53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>   54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>   55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>   56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>   57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>   58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>   59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>   60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>   61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>   62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>   63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>   64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>   65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>   66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>   67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>   68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>   69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>   70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>   71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>   72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>   73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>   74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>   75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>   76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>   77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>   78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>   79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>   80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>   81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>   82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>   83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>   84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>   85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>   86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>   87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>   88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>   89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>   90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>   91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>   92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>   93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>   94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>   95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>   96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>   97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>   98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>   99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>  100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>  101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>  102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>  103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>  104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>  105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>  106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>  107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>  108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>  109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>  110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>  111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>  112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>  113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>  114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>  115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>  116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>  117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>  118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>  119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>  120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>  121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>  122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>  123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>  124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>  125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>  126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>  127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>  128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>  129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>  130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>  131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>  132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>  133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>  134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>  135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>  136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>  137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>  138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>  139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>  140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>  141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>  142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>  143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>  144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>  145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>  146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>  147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>  148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>  149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>  150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>  151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>  152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>  153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>  154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>  155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>  156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>  157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>  158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>  159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>  160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>  161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>  162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>  163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>  164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>  165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>  166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>  167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>  168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>  169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>  170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>  171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>  172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>  173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>  174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>  175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>  176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>  177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>  178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>  179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>  180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>  181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>  182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>  183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>  184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>  185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>  186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>  187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>  188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>  189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>  190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>  191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>  192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>  193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>  194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>  195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>  196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>  197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>  198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>  199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>  200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>  201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>  202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>  203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>  204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>  205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>  206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>  207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>  208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>  209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>  210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>> 
>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>> 
>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> 
>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>   0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>   2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>> 
>>>>>>>>   0 SNES Function norm 1.132185384796e-08 
>>>>>>>>     0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>     1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>   1 SNES Function norm 2.177599365111e-12 
>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>   0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>   0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>   2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>   2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>   2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>   2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>   2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>   0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>   2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>   0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>   0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>   2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>   0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>   2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>   2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>   2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>   0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>   2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>   0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>   2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>   0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>   2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>   2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>   0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>   2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>   0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>   2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>   0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>   2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>   0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>   2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>   0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>   2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>   0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>   2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>   2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>   2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>   0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>   2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>   2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>   2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>   0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>   2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>   0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>   2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>   2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>   0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>   2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>   0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>   2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>   2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>   2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>   0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>   2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>   0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>   2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>   0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>   2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>   0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>   2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>   0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>   2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>   0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>   2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>   0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>   0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>   0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>   0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>   0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>   0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>   0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>   0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>> 
>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>> 
>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>> 
>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>> 
>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Hmm,
>>>>>>>>>>> 
>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>> 
>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>> 
>>>>>>>>>>> then 
>>>>>>>>>>> 
>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>> 
>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>> 
>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>   Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>     type: seqaij
>>>>>>>>>>>>>>>>     rows=15991, cols=15991
>>>>>>>>>>>>>>>>     package used to perform factorization: mumps
>>>>>>>>>>>>>>>>     total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>     total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>       MUMPS run parameters:
>>>>>>>>>>>>>>>>         SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>         PAR (host participation):            1 
>>>>>>>>>>>>>>>>         ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>         ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>         ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>         ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>         ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>         ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>         ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>         ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>         ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>         ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>         ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>         ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>         ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>         ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>         ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>         ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>         ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>         ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>         ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>         ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>         ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>         ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>         ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>         ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>         ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>         ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>         ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>         ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>         CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>         CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>         CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>         CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>         CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>         RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>           [0] 1.95838e+06 
>>>>>>>>>>>>>>>>         RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>           [0]  143924 
>>>>>>>>>>>>>>>>         RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>           [0]  1.95943e+06 
>>>>>>>>>>>>>>>>         INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>         [0] 7 
>>>>>>>>>>>>>>>>         INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>           [0] 7 
>>>>>>>>>>>>>>>>         INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>           [0] 15991 
>>>>>>>>>>>>>>>>         RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>         RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>         RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>         (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>         INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>         INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>         INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>         INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>         INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>         INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>         INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>         INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>         INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>         INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>         INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>         INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>         INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>         INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>         INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>         INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>         INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>         INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>         INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>         INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>         INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>         INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>         INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>         INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>         INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>         INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>         INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>         INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>         INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>     SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>     SNESSetDM()
>>>>>>>>>>>>>>>>>>     etc
>>>>>>>>>>>>>>>>>>     SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>     SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>     SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>     VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>     SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 


From gideon.simpson at gmail.com  Tue Sep  8 18:24:43 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 19:24:43 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
Message-ID: <15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>

When I use that flag, I get this error:

  0 SNES Function norm 1.421454390131e-02 
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: New nonzero at (3,0) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
[0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 19:24:08 2015
[0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
[0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: #2 MatSetValues() line 1135 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
[0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in /opt/petsc-3.5.4/src/snes/interface/snesj.c
[0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in /opt/petsc-3.5.4/src/snes/interface/snes.c
[0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /opt/petsc-3.5.4/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #6 SNESSolve() line 3743 in /opt/petsc-3.5.4/src/snes/interface/snes.c

-gideon

> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
> 
>  Barry
> 
>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>> 
>> 
>> -gideon
>> 
>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>> 
>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>> PETSc refines the DA and passes in the new one something goes wrong?
>>> 
>>>   Matt
>>> 
>>> -gideon
>>> 
>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>> 
>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>> 
>>>> Gideon,
>>>> 
>>>>  In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>> 
>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>> 
>>>> 
>>>> Barry
>>>> 
>>>>> 
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>> 
>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>> 
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>> 
>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>> 
>>>>>>>   0 SNES Function norm 3.857327250267e-09 
>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>   1 SNES Function norm 2.479697868412e-12 
>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>> 
>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>> 
>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>> 
>>>>>>>>>  0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>  1 SNES Function norm 2.177599365111e-12 
>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>> 
>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>> 
>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>> 
>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>> 
>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Hmm,
>>>>>>>>>>>> 
>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>> 
>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>> 
>>>>>>>>>>>> then 
>>>>>>>>>>>> 
>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>> 
>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>    type: seqaij
>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>      MUMPS run parameters:
>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>        PAR (host participation):            1 
>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>          [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>          [0]  143924 
>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>          [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>          [0] 15991 
>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>    SNESSetDM()
>>>>>>>>>>>>>>>>>>>    etc
>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/ee96bc41/attachment-0001.html>

From knepley at gmail.com  Tue Sep  8 19:11:27 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Sep 2015 19:11:27 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>
Message-ID: <CAMYG4G=Q6EJsJuEHpvKe0CXW2nihj8Suh7maiJpoV2d4iQ62yQ@mail.gmail.com>

On Tue, Sep 8, 2015 at 6:24 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> When I use that flag, I get this error:
>

The matrix you pass in for the Jacobian is not properly allocated. Either
set that option, or increase the allocation.

  Thanks,

    Matt


>   0 SNES Function norm 1.421454390131e-02
> *[0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------*
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
> off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named
> gs_air by gideon Tue Sep  8 19:24:08 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes
> --download-suitesparse=yes --download-superlu=yes
> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes
> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes
> --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in
> /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1135 in
> /opt/petsc-3.5.4/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in
> /opt/petsc-3.5.4/src/snes/interface/snesj.c
> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in
> /opt/petsc-3.5.4/src/snes/interface/snes.c
> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in
> /opt/petsc-3.5.4/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #6 SNESSolve() line 3743 in
> /opt/petsc-3.5.4/src/snes/interface/snes.c
>
> -gideon
>
> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>   You can run with -snes_fd it will be slow so run with only two levels of
> grid sequencing; check the convergence rate of Newton on the finer grid. It
> will hopefully be much better (also KSP should converge in one iteration).
> Send the output
>
>  Barry
>
> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> I went through my code and nothing jumps out at me as clearly wrong.  Is
> there a way to dump to disk vectors and matrices that would be useful for
> the comparison?  I imagine I?d want the jacobian that I am specifying, but
> what would I want to compare that against in terms of using
> -snes_mf_operator?
>
>
> -gideon
>
> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> Just to clarify, again, during the linear solve phase, even though the PC
> residual norm was small (getting to 10^{-10} in 2-3 iterations) that the
> true residual norm could be comparatively large signified there is
> something wrong.  The one thing i was curious about is why it seems that
> there is only an issue during grid sequencing.  I?m not seeing that kind of
> behavior on the coarsest mesh.
>
> That definitely sounds like a bug in the Jacobian evaluation. Maybe you
> assume a certain mesh size, and when
> PETSc refines the DA and passes in the new one something goes wrong?
>
>   Matt
>
> -gideon
>
> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Alright, I?ll take another look at what has been implemented in my
> Jacobian file.  One thing I?m a bit unclear about is do I need these
>  -snes_mf and snes_mf_operator flags if I have, coded in,
>
> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>
>
> Gideon,
>
>  In your case the DM creates that Jacobian matrix object and
> form_function_jacobian's job is just to fill it up with the correct values
> for the current solution value.   If you use -snes_mf_operator then SNES
> uses a matrix-free multiple for the first matrix and your Jacobian to build
> the preconditioner. With -snes_mf it uses a matrix-free multiple for the
> first and second matrix and ignores what you compute. Since it is matrix
> free in both it cannot build a preconditioner hence the -pc_type none
>
> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian
> you provide is correct
>
>
> Barry
>
>
>
> -gideon
>
> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> If the Jacobian you provide is correct then using -snes_mf with a PC type
> of LU should give two iterations (or at most three) to get very good
> (10^-12) convergence of the linear system. Here you are not getting that
> after a few SNES iterations. So I am strongly leaning to something wrong
> with the Jacobian you provide.  -snes_type test unfortunately doesn't catch
> all problems.
>
> You can try -snes_mf -pc_type none and see how the linear solver
> converges.
>
>
> Barry
>
>
>
> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Off the top of my head, I can?t rule out a null space in this problem. I
> thought I did a pretty good job of coding the Jacobian by hand.  And before
> I made that change to the Jacobian file, changing J for Jpre, if I ran with
>  -snes_check_jacobian, it appeared to be satisfactory.
>
> When I run with -ksp_type fgmres, I get
>
>   0 SNES Function norm 3.857327250267e-09
>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm
> 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm
> 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm
> 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>   1 SNES Function norm 2.479697868412e-12
> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
> 0 SNES Function norm 5.066222213176e+03
>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm
> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm
> 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
> 1 SNES Function norm 8.484696862318e+02
>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm
> 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm
> 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm
> 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
> 2 SNES Function norm 6.551139809777e+02
>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm
> 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm
> 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm
> 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
> 3 SNES Function norm 5.771483337231e+02
>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm
> 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm
> 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm
> 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
> 4 SNES Function norm 5.237540245670e+02
>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm
> 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm
> 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm
> 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
> 5 SNES Function norm 4.753058321803e+02
>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm
> 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm
> 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm
> 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
> 6 SNES Function norm 4.221482124411e+02
>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm
> 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm
> 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm
> 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
> 7 SNES Function norm 3.803620028964e+02
>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm
> 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm
> 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm
> 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
> 8 SNES Function norm 3.769519577570e+02
>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm
> 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm
> 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm
> 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm
> 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
> 9 SNES Function norm 3.766389203406e+02
>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm
> 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm
> 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm
> 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
> 10 SNES Function norm 3.751822489648e+02
>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm
> 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm
> 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm
> 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
> 11 SNES Function norm 3.737189963925e+02
>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm
> 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm
> 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm
> 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm
> 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
> 12 SNES Function norm 3.737110331594e+02
>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm
> 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm
> 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm
> 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
> 13 SNES Function norm 3.684175513154e+02
>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm
> 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm
> 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm
> 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
> 14 SNES Function norm 3.509930783933e+02
>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm
> 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm
> 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm
> 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
> 15 SNES Function norm 3.162121096565e+02
>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm
> 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm
> 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm
> 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
> 16 SNES Function norm 2.890165052818e+02
>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm
> 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm
> 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm
> 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
> 17 SNES Function norm 2.816929421445e+02
>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm
> 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm
> 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm
> 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm
> 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
> 18 SNES Function norm 2.814407559738e+02
>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm
> 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm
> 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm
> 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
> 19 SNES Function norm 2.810261821835e+02
>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm
> 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm
> 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm
> 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
> 20 SNES Function norm 2.755723672077e+02
>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm
> 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm
> 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm
> 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm
> 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm
> 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm
> 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm
> 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
> 21 SNES Function norm 2.755720389981e+02
>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm
> 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm
> 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm
> 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm
> 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm
> 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm
> 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm
> 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
> 22 SNES Function norm 2.755716464881e+02
>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm
> 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm
> 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm
> 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm
> 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm
> 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
> 23 SNES Function norm 2.755713163168e+02
>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm
> 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm
> 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm
> 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm
> 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm
> 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm
> 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm
> 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm
> 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm
> 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
> 24 SNES Function norm 2.755712363532e+02
>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm
> 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm
> 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm
> 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm
> 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm
> 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm
> 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm
> 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm
> 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm
> 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm
> 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm
> 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm
> 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm
> 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm
> 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm
> 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm
> 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm
> 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm
> 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm
> 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm
> 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm
> 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm
> 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm
> 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm
> 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm
> 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm
> 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm
> 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm
> 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm
> 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm
> 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm
> 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm
> 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm
> 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm
> 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm
> 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm
> 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm
> 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm
> 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm
> 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm
> 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm
> 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm
> 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm
> 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm
> 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm
> 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm
> 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm
> 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm
> 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm
> 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm
> 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm
> 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm
> 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm
> 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm
> 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm
> 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm
> 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm
> 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm
> 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm
> 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm
> 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm
> 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm
> 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm
> 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm
> 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm
> 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm
> 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm
> 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm
> 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm
> 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm
> 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm
> 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm
> 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm
> 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm
> 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm
> 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm
> 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm
> 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm
> 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm
> 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm
> 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm
> 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm
> 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm
> 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm
> 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm
> 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm
> 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm
> 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm
> 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm
> 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm
> 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm
> 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm
> 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm
> 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm
> 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm
> 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm
> 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm
> 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm
> 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm
> 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm
> 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm
> 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm
> 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm
> 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm
> 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm
> 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm
> 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm
> 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm
> 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm
> 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm
> 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm
> 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm
> 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm
> 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm
> 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm
> 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm
> 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm
> 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm
> 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm
> 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm
> 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm
> 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm
> 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm
> 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm
> 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm
> 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm
> 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm
> 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm
> 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm
> 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm
> 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm
> 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm
> 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm
> 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm
> 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm
> 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm
> 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm
> 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm
> 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm
> 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm
> 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm
> 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm
> 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm
> 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm
> 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm
> 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm
> 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm
> 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm
> 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm
> 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm
> 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm
> 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm
> 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm
> 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm
> 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm
> 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm
> 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm
> 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm
> 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm
> 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm
> 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm
> 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm
> 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm
> 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm
> 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm
> 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm
> 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm
> 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm
> 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm
> 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm
> 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm
> 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm
> 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm
> 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm
> 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm
> 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm
> 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm
> 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm
> 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm
> 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm
> 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm
> 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm
> 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm
> 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm
> 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm
> 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm
> 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm
> 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm
> 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm
> 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm
> 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm
> 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm
> 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm
> 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm
> 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm
> 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm
> 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm
> 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm
> 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm
> 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm
> 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm
> 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm
> 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm
> 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm
> 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm
> 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm
> 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm
> 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm
> 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm
> 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm
> 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm
> 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>
> -gideon
>
> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Is there any chance your Jacobian may have a null space? (For example with
> Neumann boundary conditions there is often a nullspace)
>
> If not run with with -ksp_type fgmres and send the same output.
>
> Something is screwy with the linear system convergence, could be several
> things including either a null space problem or that your Jacobian that you
> provide is terribly wrong.
>
> Barry
>
>
> 21 SNES Function norm 2.987064584431e+02
>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm
> 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm
> 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm
> 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>
>
> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Got it, fixing that, and returning to the original question, this is what
> I now get, when I use those two flags:
>
>  0 SNES Function norm 1.132185384796e-08
>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm
> 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm
> 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>  1 SNES Function norm 2.177599365111e-12
> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
> 0 SNES Function norm 5.066222213176e+03
>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm
> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm
> 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
> 1 SNES Function norm 8.482593852817e+02
>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm
> 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm
> 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm
> 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
> 2 SNES Function norm 6.543140468549e+02
>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm
> 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm
> 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm
> 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
> 3 SNES Function norm 5.766430557220e+02
>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm
> 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm
> 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm
> 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
> 4 SNES Function norm 5.235211958260e+02
>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm
> 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm
> 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm
> 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
> 5 SNES Function norm 4.752913229649e+02
>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm
> 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm
> 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm
> 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
> 6 SNES Function norm 4.220255380391e+02
>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm
> 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm
> 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm
> 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
> 7 SNES Function norm 3.805408907074e+02
>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm
> 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm
> 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
> 8 SNES Function norm 3.764619752339e+02
>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm
> 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm
> 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm
> 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
> 9 SNES Function norm 3.761182227091e+02
>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm
> 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm
> 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm
> 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
> 10 SNES Function norm 3.740017190063e+02
>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm
> 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm
> 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm
> 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
> 11 SNES Function norm 3.725903477238e+02
>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm
> 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm
> 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm
> 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
> 12 SNES Function norm 3.716162097231e+02
>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm
> 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm
> 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm
> 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
> 13 SNES Function norm 3.674168632847e+02
>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm
> 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm
> 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm
> 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
> 14 SNES Function norm 3.532395445266e+02
>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm
> 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm
> 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm
> 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
> 15 SNES Function norm 3.182438872366e+02
>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm
> 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm
> 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm
> 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
> 16 SNES Function norm 3.091759892779e+02
>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm
> 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm
> 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm
> 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
> 17 SNES Function norm 2.987839504359e+02
>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm
> 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm
> 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm
> 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
> 18 SNES Function norm 2.987073622777e+02
>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm
> 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm
> 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm
> 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
> 19 SNES Function norm 2.987067936734e+02
>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm
> 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm
> 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm
> 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
> 20 SNES Function norm 2.987067502910e+02
>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm
> 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm
> 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm
> 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
> 21 SNES Function norm 2.987064584431e+02
>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm
> 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm
> 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm
> 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
> 22 SNES Function norm 2.987064525262e+02
>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm
> 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm
> 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm
> 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
> 23 SNES Function norm 2.987064121622e+02
>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm
> 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm
> 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm
> 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
> 24 SNES Function norm 2.987063973426e+02
>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm
> 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm
> 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm
> 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
> 25 SNES Function norm 2.987063920553e+02
>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm
> 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm
> 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm
> 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
> 26 SNES Function norm 2.987063919786e+02
>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm
> 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm
> 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm
> 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
> 27 SNES Function norm 2.987063393145e+02
>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm
> 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm
> 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm
> 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
> 28 SNES Function norm 2.987063253277e+02
>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm
> 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm
> 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm
> 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
> 29 SNES Function norm 2.987063197153e+02
>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm
> 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm
> 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm
> 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
> 30 SNES Function norm 2.987063193089e+02
>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm
> 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm
> 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm
> 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
> 31 SNES Function norm 2.987063192061e+02
>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm
> 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm
> 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm
> 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
> 32 SNES Function norm 2.987063094492e+02
>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm
> 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm
> 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm
> 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
> 33 SNES Function norm 2.987063089942e+02
>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm
> 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm
> 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm
> 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
> 34 SNES Function norm 2.987062660109e+02
>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm
> 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm
> 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm
> 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
> 35 SNES Function norm 2.987062107900e+02
>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm
> 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm
> 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm
> 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
> 36 SNES Function norm 2.987062055224e+02
>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm
> 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm
> 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm
> 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
> 37 SNES Function norm 2.987061774798e+02
>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm
> 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm
> 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm
> 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
> 38 SNES Function norm 2.987061715400e+02
>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm
> 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm
> 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm
> 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
> 39 SNES Function norm 2.987061699634e+02
>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm
> 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm
> 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm
> 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
> 40 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm
> 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
> 41 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm
> 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
> 42 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm
> 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
> 43 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm
> 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
> 44 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm
> 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
> 45 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm
> 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
> 46 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm
> 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
> 47 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm
> 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>
>
> -gideon
>
> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> This indicates that somewhere in your ComputeJacobian you are setting
> matrix entries with the first Mat argument when you should always set them
> with the second matrix argument. For example if you have
>
> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>
> you should call all the MatSetValues() with jpre, no J. Then at the end of
> the function you should call MatAssemblyBegin/End() on jpre then on J if J
> is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>
> This is a minor glitch we'll get past.
>
> Barry
>
> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> By the way, I tried using a different petsc installation, and now, rather
> than the segmentation fault, I get the following error:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Mat type mffd
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named
> gs_air by gideon Mon Sep  7 21:32:18 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes
> --download-suitesparse=yes --download-superlu=yes
> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes
> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes
> --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in
> /opt/petsc-3.5.4/src/mat/interface/matrix.c
>
> -gideon
>
> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Hmm,
>
> Ok you can try running it directly in the debugger since it is one
> process, type
>
> gdb ./blowup_batch_refine
>
> then
>
> when the debugger comes up (if it does not cut and paste all output and
> send it)
>
> run -on_error_abort -snes_mf_operator  and any other options you normally
> use
>
>
> Barry
>
> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Running with that flag gives me this:
>
> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111
> on gs_air
> Unable to start debugger: No such file or directory
>
>
>
> -gideon
>
> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> This should not happen. Run with a debug version of PETSc installed and
> the option -start_in_debugger noxterm  Once the debugger starts up type
> cont and when it crashes type where or bt  Send all output
>
>
>
> Barry
>
>
> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> I?m getting an error with -snes_mf_operator,
>
> 0 SNES Function norm 1.421454390131e-02
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by
> gideon Mon Sep  7 21:08:19 2015
> [0]PETSC ERROR: Configure options --prefix=/opt/local
> --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries
> --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0
> --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate
> --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local
> --with-superlu-dir=/opt/local --with-metis-dir=/opt/local
> --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local
> --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local
> CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp
> FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp
> F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os
> FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names"
> CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os
> FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports
> --with-mpiexec=mpiexec-mpich-mp
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>
> -gideon
>
> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc
> SNES is generating a poor descent direction. You can try
> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What
> happens?
>
> Barry
>
>
>
> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> No problem Matt, I don?t think we had previously discussed that output.
> Here is a case where things fail.
>
> 0 SNES Function norm 4.027481756921e-09
> 1 SNES Function norm 1.760477878365e-12
> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
> 0 SNES Function norm 5.066222213176e+03
> 1 SNES Function norm 8.484697184230e+02
> 2 SNES Function norm 6.549559723294e+02
> 3 SNES Function norm 5.770723278153e+02
> 4 SNES Function norm 5.237702240594e+02
> 5 SNES Function norm 4.753909019848e+02
> 6 SNES Function norm 4.221784590755e+02
> 7 SNES Function norm 3.806525080483e+02
> 8 SNES Function norm 3.762054656019e+02
> 9 SNES Function norm 3.758975226873e+02
> 10 SNES Function norm 3.757032042706e+02
> 11 SNES Function norm 3.728798164234e+02
> 12 SNES Function norm 3.723078741075e+02
> 13 SNES Function norm 3.721848059825e+02
> 14 SNES Function norm 3.720227575629e+02
> 15 SNES Function norm 3.720051998555e+02
> 16 SNES Function norm 3.718945430587e+02
> 17 SNES Function norm 3.700412694044e+02
> 18 SNES Function norm 3.351964889461e+02
> 19 SNES Function norm 3.096016086233e+02
> 20 SNES Function norm 3.008410789787e+02
> 21 SNES Function norm 2.752316716557e+02
> 22 SNES Function norm 2.707658474165e+02
> 23 SNES Function norm 2.698436736049e+02
> 24 SNES Function norm 2.618233857172e+02
> 25 SNES Function norm 2.600121920634e+02
> 26 SNES Function norm 2.585046423168e+02
> 27 SNES Function norm 2.568551090220e+02
> 28 SNES Function norm 2.556404537064e+02
> 29 SNES Function norm 2.536353523683e+02
> 30 SNES Function norm 2.533596070171e+02
> 31 SNES Function norm 2.532324379596e+02
> 32 SNES Function norm 2.531842335211e+02
> 33 SNES Function norm 2.531684527520e+02
> 34 SNES Function norm 2.531637604618e+02
> 35 SNES Function norm 2.531624767821e+02
> 36 SNES Function norm 2.531621359093e+02
> 37 SNES Function norm 2.531620504925e+02
> 38 SNES Function norm 2.531620350055e+02
> 39 SNES Function norm 2.531620310522e+02
> 40 SNES Function norm 2.531620300471e+02
> 41 SNES Function norm 2.531620298084e+02
> 42 SNES Function norm 2.531620297478e+02
> 43 SNES Function norm 2.531620297324e+02
> 44 SNES Function norm 2.531620297303e+02
> 45 SNES Function norm 2.531620297302e+02
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
> 0 SNES Function norm 9.636339304380e+03
> 1 SNES Function norm 8.997731184634e+03
> 2 SNES Function norm 8.120498349232e+03
> 3 SNES Function norm 7.322379894820e+03
> 4 SNES Function norm 6.599581599149e+03
> 5 SNES Function norm 6.374872854688e+03
> 6 SNES Function norm 6.372518007653e+03
> 7 SNES Function norm 6.073996314301e+03
> 8 SNES Function norm 5.635965277054e+03
> 9 SNES Function norm 5.155389064046e+03
> 10 SNES Function norm 5.080567902638e+03
> 11 SNES Function norm 5.058878643969e+03
> 12 SNES Function norm 5.058835649793e+03
> 13 SNES Function norm 5.058491285707e+03
> 14 SNES Function norm 5.057452865337e+03
> 15 SNES Function norm 5.057226140688e+03
> 16 SNES Function norm 5.056651272898e+03
> 17 SNES Function norm 5.056575190057e+03
> 18 SNES Function norm 5.056574632598e+03
> 19 SNES Function norm 5.056574520229e+03
> 20 SNES Function norm 5.056574492569e+03
> 21 SNES Function norm 5.056574485124e+03
> 22 SNES Function norm 5.056574483029e+03
> 23 SNES Function norm 5.056574482427e+03
> 24 SNES Function norm 5.056574482302e+03
> 25 SNES Function norm 5.056574482287e+03
> 26 SNES Function norm 5.056574482282e+03
> 27 SNES Function norm 5.056574482281e+03
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
> SNES Object: 1 MPI processes
> type: newtonls
> maximum iterations=50, maximum function evaluations=10000
> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
> total number of linear solver iterations=28
> total number of function evaluations=323
> total number of grid sequence refinements=2
> SNESLineSearch Object:   1 MPI processes
> type: bt
> interpolation: cubic
> alpha=1.000000e-04
> maxstep=1.000000e+08, minlambda=1.000000e-12
> tolerances: relative=1.000000e-08, absolute=1.000000e-15,
> lambda=1.000000e-08
> maximum iterations=40
> KSP Object:   1 MPI processes
> type: gmres
> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=10000, initial guess is zero
> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using PRECONDITIONED norm type for convergence test
> PC Object:   1 MPI processes
> type: lu
> LU: out-of-place factorization
> tolerance for zero pivot 2.22045e-14
> matrix ordering: nd
> factor fill ratio given 0, needed 0
> Factored matrix follows:
>  Mat Object:           1 MPI processes
>    type: seqaij
>    rows=15991, cols=15991
>    package used to perform factorization: mumps
>    total: nonzeros=255801, allocated nonzeros=255801
>    total number of mallocs used during MatSetValues calls =0
>      MUMPS run parameters:
>        SYM (matrix type):                   0
>        PAR (host participation):            1
>        ICNTL(1) (output for error):         6
>        ICNTL(2) (output of diagnostic msg): 0
>        ICNTL(3) (output for global info):   0
>        ICNTL(4) (level of printing):        0
>        ICNTL(5) (input mat struct):         0
>        ICNTL(6) (matrix prescaling):        7
>        ICNTL(7) (sequentia matrix ordering):6
>        ICNTL(8) (scalling strategy):        77
>        ICNTL(10) (max num of refinements):  0
>        ICNTL(11) (error analysis):          0
>        ICNTL(12) (efficiency control):                         1
>        ICNTL(13) (efficiency control):                         0
>        ICNTL(14) (percentage of estimated workspace increase): 20
>        ICNTL(18) (input mat struct):                           0
>        ICNTL(19) (Shur complement info):                       0
>        ICNTL(20) (rhs sparse pattern):                         0
>        ICNTL(21) (somumpstion struct):                            0
>        ICNTL(22) (in-core/out-of-core facility):               0
>        ICNTL(23) (max size of memory can be allocated locally):0
>        ICNTL(24) (detection of null pivot rows):               0
>        ICNTL(25) (computation of a null space basis):          0
>        ICNTL(26) (Schur options for rhs or solution):          0
>        ICNTL(27) (experimental parameter):                     -8
>        ICNTL(28) (use parallel or sequential ordering):        1
>        ICNTL(29) (parallel ordering):                          0
>        ICNTL(30) (user-specified set of entries in inv(A)):    0
>        ICNTL(31) (factors is discarded in the solve phase):    0
>        ICNTL(33) (compute determinant):                        0
>        CNTL(1) (relative pivoting threshold):      0.01
>        CNTL(2) (stopping criterion of refinement): 1.49012e-08
>        CNTL(3) (absomumpste pivoting threshold):      0
>        CNTL(4) (vamumpse of static pivoting):         -1
>        CNTL(5) (fixation for null pivots):         0
>        RINFO(1) (local estimated flops for the elimination after
> analysis):
>          [0] 1.95838e+06
>        RINFO(2) (local estimated flops for the assembly after
> factorization):
>          [0]  143924
>        RINFO(3) (local estimated flops for the elimination after
> factorization):
>          [0]  1.95943e+06
>        INFO(15) (estimated size of (in MB) MUMPS internal data for running
> numerical factorization):
>        [0] 7
>        INFO(16) (size of (in MB) MUMPS internal data used during numerical
> factorization):
>          [0] 7
>        INFO(23) (num of pivots eliminated on this processor after
> factorization):
>          [0] 15991
>        RINFOG(1) (global estimated flops for the elimination after
> analysis): 1.95838e+06
>        RINFOG(2) (global estimated flops for the assembly after
> factorization): 143924
>        RINFOG(3) (global estimated flops for the elimination after
> factorization): 1.95943e+06
>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>        INFOG(3) (estimated real workspace for factors on all processors
> after analysis): 255801
>        INFOG(4) (estimated integer workspace for factors on all processors
> after analysis): 127874
>        INFOG(5) (estimated maximum front size in the complete tree): 11
>        INFOG(6) (number of nodes in the complete tree): 3996
>        INFOG(7) (ordering option effectively use after analysis): 6
>        INFOG(8) (structural symmetry in percent of the permuted matrix
> after analysis): 86
>        INFOG(9) (total real/complex workspace to store the matrix factors
> after factorization): 255865
>        INFOG(10) (total integer space store the matrix factors after
> factorization): 127890
>        INFOG(11) (order of largest frontal matrix after factorization): 11
>        INFOG(12) (number of off-diagonal pivots): 19
>        INFOG(13) (number of delayed pivots after factorization): 8
>        INFOG(14) (number of memory compress after factorization): 0
>        INFOG(15) (number of steps of iterative refinement after solution):
> 0
>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for
> factorization after analysis: value on the most memory consuming
> processor): 7
>        INFOG(17) (estimated size of all MUMPS internal data for
> factorization after analysis: sum over all processors): 7
>        INFOG(18) (size of all MUMPS internal data allocated during
> factorization: value on the most memory consuming processor): 7
>        INFOG(19) (size of all MUMPS internal data allocated during
> factorization: sum over all processors): 7
>        INFOG(20) (estimated number of entries in the factors): 255801
>        INFOG(21) (size in MB of memory effectively used during
> factorization - value on the most memory consuming processor): 7
>        INFOG(22) (size in MB of memory effectively used during
> factorization - sum over all processors): 7
>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0
>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1
>        INFOG(25) (after factorization: number of pivots modified by static
> pivoting): 0
>        INFOG(28) (after factorization: number of null pivots encountered):
> 0
>        INFOG(29) (after factorization: effective number of entries in the
> factors (sum over all processors)): 255865
>        INFOG(30, 31) (after solution: size in Mbytes of memory used during
> solution phase): 5, 5
>        INFOG(32) (after analysis: type of analysis done): 1
>        INFOG(33) (value used for ICNTL(8)): 7
>        INFOG(34) (exponent of the determinant if determinant is
> requested): 0
> linear system matrix = precond matrix:
> Mat Object:     1 MPI processes
> type: seqaij
> rows=15991, cols=15991
> total: nonzeros=223820, allocated nonzeros=431698
> total number of mallocs used during MatSetValues calls =15991
> using I-node routines: found 4000 nodes, limit used is 5
>
>
>
>
> -gideon
>
> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> Barry,
>
> I finally got a chance to really try using the grid sequencing within my
> code.  I find that, in some cases, even if it can solve successfully on the
> coarsest mesh, the SNES fails, usually due to a line search failure, when
> it tries to compute along the grid sequence.  Would you have any
> suggestions?
>
> I apologize if I have asked before, but can you give me -snes_view for the
> solver? I could not find it in the email thread.
>
> I would suggest trying to fiddle with the line search, or precondition it
> with Richardson. It would be nice to see -snes_monitor
> for the runs that fail, and then we can break down the residual into
> fields and look at it again (if my custom residual monitor
> does not work we can write one easily). Seeing which part of the residual
> does not converge is key to designing the NASM
> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai,
> present it. We need better monitoring in PETSc.
>
> Thanks,
>
> Matt
>
> -gideon
>
> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Yes, if i continue in this parameter on the coarse mesh, I can generally
> solve at all values. I do find that I need to do some amount of
> continuation to solve near the endpoint.  The problem is that on the coarse
> mesh, things are not fully resolved at all the values along the
> continuation parameter, and I would like to do refinement.
>
> One subtlety is that I actually want the intermediate continuation
> solutions  too.  Currently, without doing any grid sequence, I compute
> each, write it to disk, and then go on to the next one.  So I now need to
> go back an refine them.  I was thinking that perhaps I could refine them on
> the fly, dump them to disk, and use the coarse solution as the starting
> guess at the next iteration, but that would seem to require resetting the
> snes back to the coarse grid.
>
> The alternative would be to just script the mesh refinement in a post
> processing stage, where each value of the continuation is parameter is
> loaded on the coarse mesh, and refined.  Perhaps that?s the most practical
> thing to do.
>
>
> I would do the following. Create your DM and create a SNES that will do
> the continuation
>
> loop over continuation parameter
>
> SNESSolve(snes,NULL,Ucoarse);
>
> if (you decide you want to see the refined solution at this continuation
> point) {
>    SNESCreate(comm,&snesrefine);
>    SNESSetDM()
>    etc
>    SNESSetGridSequence(snesrefine,)
>    SNESSolve(snesrefine,0,Ucoarse);
>    SNESGetSolution(snesrefine,&Ufine);
>    VecView(Ufine or do whatever you want to do with the Ufine at that
> continuation point
>    SNESDestroy(snesrefine);
> end if
>
> end loop over continuation parameter.
>
> Barry
>
>
> -gideon
>
> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>
> 3.  This problem is actually part of a continuation problem that roughly
> looks like this
>
> for( continuation parameter p = 0 to 1){
>
> solve with parameter p_i using solution from p_{i-1},
> }
>
> What I would like to do is to start the solver, for each value of
> parameter p_i on the coarse mesh, and then do grid sequencing on that.  But
> it appears that after doing grid sequencing on the initial p_0 = 0, the
> SNES is set to use the finer mesh.
>
>
> So you are using continuation to give you a good enough initial guess on
> the coarse level to even get convergence on the coarse level? First I would
> check if you even need the continuation (or can you not even solve the
> coarse problem without it).
>
> If you do need the continuation then you will need to tweak how you do the
> grid sequencing. I think this will work:
>
> Do not use -snes_grid_sequencing
>
> Run SNESSolve() as many times as you want with your continuation
> parameter. This will all happen on the coarse mesh.
>
> Call SNESSetGridSequence()
>
> Then call SNESSolve() again and it will do one solve on the coarse level
> and then interpolate to the next level etc.
>
>
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/ba19a7d0/attachment-0001.html>

From gideon.simpson at gmail.com  Tue Sep  8 19:15:54 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 20:15:54 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAMYG4G=Q6EJsJuEHpvKe0CXW2nihj8Suh7maiJpoV2d4iQ62yQ@mail.gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>
	<CAMYG4G=Q6EJsJuEHpvKe0CXW2nihj8Suh7maiJpoV2d4iQ62yQ@mail.gmail.com>
Message-ID: <F70D32BE-14C5-4631-8139-1ED02FC4A43D@gmail.com>

I had been using

SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);

Do I need to manually set the matrices to use that flag?

-gideon

> On Sep 8, 2015, at 8:11 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Sep 8, 2015 at 6:24 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> When I use that flag, I get this error:
> 
> The matrix you pass in for the Jacobian is not properly allocated. Either set that option, or increase the allocation.
> 
>   Thanks,
> 
>     Matt
>  
>   0 SNES Function norm 1.421454390131e-02 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 19:24:08 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1135 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in /opt/petsc-3.5.4/src/snes/interface/snesj.c
> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in /opt/petsc-3.5.4/src/snes/interface/snes.c
> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /opt/petsc-3.5.4/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /opt/petsc-3.5.4/src/snes/interface/snes.c
> 
> -gideon
> 
>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>> 
>> 
>>   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>> 
>>  Barry
>> 
>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>> 
>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>> 
>>> 
>>> -gideon
>>> 
>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>> 
>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>> 
>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>> 
>>>>   Matt
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>> 
>>>>> 
>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>> 
>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>> 
>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>> 
>>>>> Gideon,
>>>>> 
>>>>>  In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>> 
>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>> 
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>> 
>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>> 
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>> 
>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>> 
>>>>>>>>   0 SNES Function norm 3.857327250267e-09 
>>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>   1 SNES Function norm 2.479697868412e-12 
>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>> 
>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>> 
>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>> 
>>>>>>>>>>  0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>  1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>> 
>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>> 
>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>> 
>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>> 
>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>> 
>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>> 
>>>>>>>>>>>>> then 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>    type: seqaij
>>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>      MUMPS run parameters:
>>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>        PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>          [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>          [0]  143924 
>>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>          [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>          [0] 15991 
>>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>    SNESSetDM()
>>>>>>>>>>>>>>>>>>>>    etc
>>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/81face13/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Sep  8 19:56:50 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 Sep 2015 19:56:50 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>
Message-ID: <EEA729D5-6242-4071-A8E1-9B0A752CAF24@mcs.anl.gov>


  This is our fault. You need to add a line 

  ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);CHKERRQ(ierr);

after 

  PetscFunctionBegin;

in SNESComputeJacobianDefault()

  then do

make gnumake 

in that directory.

  You really should update to PETSc 3.6 (http://www.mcs.anl.gov/petsc/download/index.html note it is best to obtain with git) It will make both your life and our lives easier and there is no downside.

  Barry

> On Sep 8, 2015, at 6:24 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> When I use that flag, I get this error:
> 
>   0 SNES Function norm 1.421454390131e-02 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 19:24:08 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1135 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in /opt/petsc-3.5.4/src/snes/interface/snesj.c
> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in /opt/petsc-3.5.4/src/snes/interface/snes.c
> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /opt/petsc-3.5.4/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /opt/petsc-3.5.4/src/snes/interface/snes.c
> 
> -gideon
> 
>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>> 
>>  Barry
>> 
>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>> 
>>> 
>>> -gideon
>>> 
>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>> 
>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>> 
>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>> 
>>>>   Matt
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>> 
>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>> 
>>>>> Gideon,
>>>>> 
>>>>>  In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>> 
>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>> 
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>> 
>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>> 
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>> 
>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>> 
>>>>>>>>   0 SNES Function norm 3.857327250267e-09 
>>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>   1 SNES Function norm 2.479697868412e-12 
>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>> 
>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>> 
>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>> 
>>>>>>>>>>  0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>  1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>> 
>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>> 
>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>> 
>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>> 
>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>> 
>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>> 
>>>>>>>>>>>>> then 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>    type: seqaij
>>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>      MUMPS run parameters:
>>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>        PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>          [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>          [0]  143924 
>>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>          [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>          [0] 15991 
>>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>    SNESSetDM()
>>>>>>>>>>>>>>>>>>>>    etc
>>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>> 
> 


From gideon.simpson at gmail.com  Tue Sep  8 20:23:25 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 21:23:25 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <EEA729D5-6242-4071-A8E1-9B0A752CAF24@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>
	<EEA729D5-6242-4071-A8E1-9B0A752CAF24@mcs.anl.gov>
Message-ID: <9DF2724E-BB51-4892-9E3D-DB6D7B286C39@gmail.com>

I just installed a fresh 3.6.1 pull, compiled against it, and I?m still having an error.
  0 SNES Function norm 1.421454390131e-02 
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: New nonzero at (3,0) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
[0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 21:21:11 2015
[0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
[0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 485 in /opt/petsc/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: #2 MatSetValues() line 1173 in /opt/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 121 in /opt/petsc/src/snes/interface/snesj.c
[0]PETSC ERROR: #4 SNESComputeJacobian() line 2232 in /opt/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 231 in /opt/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #6 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR:  Vec is locked read only, argument # 1
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
[0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 21:21:11 2015
[0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
[0]PETSC ERROR: #7 VecGetArray() line 1646 in /opt/petsc/src/vec/vec/interface/rvector.c




-gideon

> On Sep 8, 2015, at 8:56 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  This is our fault. You need to add a line 
> 
>  ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);CHKERRQ(ierr);
> 
> after 
> 
>  PetscFunctionBegin;
> 
> in SNESComputeJacobianDefault()
> 
>  then do
> 
> make gnumake 
> 
> in that directory.
> 
>  You really should update to PETSc 3.6 (http://www.mcs.anl.gov/petsc/download/index.html note it is best to obtain with git) It will make both your life and our lives easier and there is no downside.
> 
>  Barry
> 
>> On Sep 8, 2015, at 6:24 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> When I use that flag, I get this error:
>> 
>>  0 SNES Function norm 1.421454390131e-02 
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Argument out of range
>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 19:24:08 2015
>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c
>> [0]PETSC ERROR: #2 MatSetValues() line 1135 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in /opt/petsc-3.5.4/src/snes/interface/snesj.c
>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /opt/petsc-3.5.4/src/snes/impls/ls/ls.c
>> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>> 
>> -gideon
>> 
>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>>  You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>> 
>>> Barry
>>> 
>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>> 
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>> 
>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>>> 
>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>> 
>>>>>  Matt
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>> 
>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>> 
>>>>>> Gideon,
>>>>>> 
>>>>>> In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>> 
>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>> 
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>>> 
>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>> 
>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>> 
>>>>>>>>>  0 SNES Function norm 3.857327250267e-09 
>>>>>>>>>    0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>    1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>    2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>  1 SNES Function norm 2.479697868412e-12 
>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>  0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>  2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>  3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>  2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>  2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>  3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>  2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>  2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>  2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>  2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>  3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>  2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>  2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>  2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>  3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>  4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>  5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>  6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>  2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>  3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>  4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>  5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>  6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>  2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>  3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>  4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>  2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>  3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>  4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>  5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>  6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>  7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>  8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>>  0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>  2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>  3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>  4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>  5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>  6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>  7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>  8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>  9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>> 10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>> 11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>> 12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>> 13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>> 14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>> 15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>> 16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>> 17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>> 18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>> 19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>> 20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>> 21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>> 22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>> 23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>> 24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>> 25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>> 26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>> 27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>> 28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>> 29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>> 30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>> 31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>> 32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>> 33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>> 34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>> 35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>> 36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>> 37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>> 38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>> 39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>> 40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>> 41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>> 42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>> 43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>> 44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>> 45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>> 46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>> 47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>> 48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>> 49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>> 50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>> 51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>> 52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>> 53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>> 54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>> 55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>> 56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>> 57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>> 58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>> 59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>> 60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>> 61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>> 62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>> 63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>> 64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>> 65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>> 66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>> 67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>> 68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>> 69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>> 70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>> 71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>> 72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>> 73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>> 74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>> 75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>> 76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>> 77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>> 78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>> 79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>> 80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>> 81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>> 82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>> 83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>> 84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>> 85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>> 86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>> 87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>> 88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>> 89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>> 90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>> 91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>> 92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>> 93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>> 94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>> 95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>> 96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>> 97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>> 98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>> 99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>> 
>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>> 
>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>> 
>>>>>>>>>>> 0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>>   0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>> 1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>> 0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>> 2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>> 2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>> 2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>> 2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>> 2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>> 2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>> 2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>> 2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>> 2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>> 2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>> 2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>> 2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>> 2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>> 2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>> 2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>> 
>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>> 
>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>> 
>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> then 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>> Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>>   type: seqaij
>>>>>>>>>>>>>>>>>>>   rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>   package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>   total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>   total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>     MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>       SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>>       PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>>       ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>>       ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>>       ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>>       ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>>       ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>>       ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>>       ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>>       ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>>       ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>>       ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>>       ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>>       ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>>       ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>>       ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>>       ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>>       ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>>       ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>>       ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>>       ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>>       ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>>       ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>>       ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>>       ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>>       ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>>       ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>>       ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>>       ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>>       ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>>       CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>>       CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>>       CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>>       CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>>       CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>>       RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>>         [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>>       RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>>         [0]  143924 
>>>>>>>>>>>>>>>>>>>       RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>>         [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>>       INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>>       [0] 7 
>>>>>>>>>>>>>>>>>>>       INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>>         [0] 7 
>>>>>>>>>>>>>>>>>>>       INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>>         [0] 15991 
>>>>>>>>>>>>>>>>>>>       RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>>       RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>>       RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>>       (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>       INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>>       INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>>       INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>>       INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>>       INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>>       INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>>       INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>>       INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>>       INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>>       INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>>       INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>>       INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>>       INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>>       INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>       INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>       INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>       INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>       INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>>       INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>       INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>       INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>>       INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>>       INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>>       INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>       INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>       INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>       INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>       INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>       INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>   SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>   SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>   etc
>>>>>>>>>>>>>>>>>>>>>   SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>   SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>   SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>   VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>   SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/5f14d307/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Sep  8 20:27:18 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 Sep 2015 20:27:18 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <9DF2724E-BB51-4892-9E3D-DB6D7B286C39@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>
	<EEA729D5-6242-4071-A8E1-9B0A752CAF24@mcs.anl.gov>
	<9DF2724E-BB51-4892-9E3D-DB6D7B286C39@gmail.com>
Message-ID: <5B5109E8-6C37-4735-ACCE-54ED7C18E218@mcs.anl.gov>


  Did you add the line I told you to?


> On Sep 8, 2015, at 8:23 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I just installed a fresh 3.6.1 pull, compiled against it, and I?m still having an error.
>   0 SNES Function norm 1.421454390131e-02 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 21:21:11 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 485 in /opt/petsc/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1173 in /opt/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 121 in /opt/petsc/src/snes/interface/snesj.c
> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2232 in /opt/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 231 in /opt/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #6 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR:  Vec is locked read only, argument # 1
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 21:21:11 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #7 VecGetArray() line 1646 in /opt/petsc/src/vec/vec/interface/rvector.c
> 
> 
> 
> 
> -gideon
> 
>> On Sep 8, 2015, at 8:56 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  This is our fault. You need to add a line 
>> 
>>  ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);CHKERRQ(ierr);
>> 
>> after 
>> 
>>  PetscFunctionBegin;
>> 
>> in SNESComputeJacobianDefault()
>> 
>>  then do
>> 
>> make gnumake 
>> 
>> in that directory.
>> 
>>  You really should update to PETSc 3.6 (http://www.mcs.anl.gov/petsc/download/index.html note it is best to obtain with git) It will make both your life and our lives easier and there is no downside.
>> 
>>  Barry
>> 
>>> On Sep 8, 2015, at 6:24 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> When I use that flag, I get this error:
>>> 
>>>  0 SNES Function norm 1.421454390131e-02 
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Argument out of range
>>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 19:24:08 2015
>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c
>>> [0]PETSC ERROR: #2 MatSetValues() line 1135 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in /opt/petsc-3.5.4/src/snes/interface/snesj.c
>>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /opt/petsc-3.5.4/src/snes/impls/ls/ls.c
>>> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>>> 
>>> -gideon
>>> 
>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>>  You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>>> 
>>>> Barry
>>>> 
>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>>> 
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>> 
>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>>>> 
>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>>> 
>>>>>>  Matt
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>>> 
>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>>> 
>>>>>>> Gideon,
>>>>>>> 
>>>>>>> In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>>> 
>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>>> 
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>>>> 
>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>>> 
>>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>>> 
>>>>>>>>>>  0 SNES Function norm 3.857327250267e-09 
>>>>>>>>>>    0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>    1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>>    2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>>  1 SNES Function norm 2.479697868412e-12 
>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>>  2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>>  3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>>  2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>>  2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>>  3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>>  2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>>  2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>>  2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>>  2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>>  3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>>  2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>>  2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>>  2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>>  3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>>  4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>>  5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>>  6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>>  2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>>  3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>>  4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>>  5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>>  6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>>  2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>>  3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>>  4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>>  2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>>  3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>>  4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>>  5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>>  6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>>  7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>>  8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>>>  0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>>  2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>>  3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>>  4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>>  5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>>  6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>>  7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>>  8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>>  9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>>> 10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>>> 11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>>> 12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>>> 13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>>> 14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>>> 15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>>> 16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>>> 17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>>> 18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>>> 19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>>> 20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>>> 21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>>> 22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>>> 23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>>> 24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>>> 25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>>> 26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>>> 27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>>> 28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>>> 29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>>> 30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>>> 31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>>> 32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>>> 33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>>> 34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>>> 35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>>> 36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>>> 37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>>> 38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>>> 39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>>> 40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>>> 41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>>> 42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>>> 43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>>> 44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>>> 45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>>> 46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>>> 47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>>> 48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>>> 49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>>> 50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>>> 51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>>> 52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>>> 53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>>> 54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>>> 55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>>> 56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>>> 57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>>> 58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>>> 59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>>> 60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>>> 61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>>> 62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>>> 63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>>> 64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>>> 65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>>> 66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>>> 67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>>> 68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>>> 69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>>> 70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>>> 71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>>> 72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>>> 73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>>> 74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>>> 75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>>> 76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>>> 77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>>> 78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>>> 79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>>> 80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>>> 81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>>> 82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>>> 83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>>> 84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>>> 85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>>> 86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>>> 87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>>> 88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>>> 89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>>> 90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>>> 91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>>> 92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>>> 93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>>> 94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>>> 95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>>> 96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>>> 97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>>> 98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>>> 99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>>> 
>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>>> 
>>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>>> 
>>>>>>>>>>>> 0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>>>   0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>>> 1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>>> 
>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> then 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>>> Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>>>   type: seqaij
>>>>>>>>>>>>>>>>>>>>   rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>   package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>>   total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>>   total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>>     MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>>       SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>>>       PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>>>       ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>>>       ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>>>       ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>>>       ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>>>       ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>>>       ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>>>       ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>>>       ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>>>       ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>>>       ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>>>       CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>>>       CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>>>       CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>>>       CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>>>       CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>>>       RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>>>         [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>       RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>>>         [0]  143924 
>>>>>>>>>>>>>>>>>>>>       RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>>>         [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>>>       INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>>>       [0] 7 
>>>>>>>>>>>>>>>>>>>>       INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>>>         [0] 7 
>>>>>>>>>>>>>>>>>>>>       INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>>>         [0] 15991 
>>>>>>>>>>>>>>>>>>>>       RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>       RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>>>       RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>>>       (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>>       INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>>>       INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>>>       INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>>>       INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>>>       INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>>>       INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>>>       INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>>>       INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>>>       INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>>>       INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>>>       INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>>>       INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>>>       INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>>>       INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>       INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>       INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>       INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>       INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>>>       INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>       INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>       INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>>>       INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>>>       INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>>>       INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>>       INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>>       INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>>       INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>>       INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>>       INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>>   SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>>   SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>>   etc
>>>>>>>>>>>>>>>>>>>>>>   SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>>   SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>   SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>>   VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>>   SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>> 
>>>> 
>>> 
>> 
> 


From gideon.simpson at gmail.com  Tue Sep  8 20:32:08 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 21:32:08 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <5B5109E8-6C37-4735-ACCE-54ED7C18E218@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>
	<EEA729D5-6242-4071-A8E1-9B0A752CAF24@mcs.anl.gov>
	<9DF2724E-BB51-4892-9E3D-DB6D7B286C39@gmail.com>
	<5B5109E8-6C37-4735-ACCE-54ED7C18E218@mcs.anl.gov>
Message-ID: <F30D7B58-9EFB-4BFF-AA4A-64605586B043@gmail.com>

The message was unclear.  I assumed that you meant if I went up to 3.6.1, this would be corrected.  What file am I editing and in which directory?


-gideon

> On Sep 8, 2015, at 9:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Did you add the line I told you to?
> 
> 
>> On Sep 8, 2015, at 8:23 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> I just installed a fresh 3.6.1 pull, compiled against it, and I?m still having an error.
>>  0 SNES Function norm 1.421454390131e-02 
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Argument out of range
>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 21:21:11 2015
>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 485 in /opt/petsc/src/mat/impls/aij/seq/aij.c
>> [0]PETSC ERROR: #2 MatSetValues() line 1173 in /opt/petsc/src/mat/interface/matrix.c
>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 121 in /opt/petsc/src/snes/interface/snesj.c
>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2232 in /opt/petsc/src/snes/interface/snes.c
>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 231 in /opt/petsc/src/snes/impls/ls/ls.c
>> [0]PETSC ERROR: #6 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Object is in wrong state
>> [0]PETSC ERROR:  Vec is locked read only, argument # 1
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 21:21:11 2015
>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>> [0]PETSC ERROR: #7 VecGetArray() line 1646 in /opt/petsc/src/vec/vec/interface/rvector.c
>> 
>> 
>> 
>> 
>> -gideon
>> 
>>> On Sep 8, 2015, at 8:56 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> This is our fault. You need to add a line 
>>> 
>>> ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);CHKERRQ(ierr);
>>> 
>>> after 
>>> 
>>> PetscFunctionBegin;
>>> 
>>> in SNESComputeJacobianDefault()
>>> 
>>> then do
>>> 
>>> make gnumake 
>>> 
>>> in that directory.
>>> 
>>> You really should update to PETSc 3.6 (http://www.mcs.anl.gov/petsc/download/index.html note it is best to obtain with git) It will make both your life and our lives easier and there is no downside.
>>> 
>>> Barry
>>> 
>>>> On Sep 8, 2015, at 6:24 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> When I use that flag, I get this error:
>>>> 
>>>> 0 SNES Function norm 1.421454390131e-02 
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: Argument out of range
>>>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
>>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 19:24:08 2015
>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c
>>>> [0]PETSC ERROR: #2 MatSetValues() line 1135 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in /opt/petsc-3.5.4/src/snes/interface/snesj.c
>>>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>>>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /opt/petsc-3.5.4/src/snes/impls/ls/ls.c
>>>> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>> 
>>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>>>>> 
>>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>>>> 
>>>>>>> Matt
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>>>> 
>>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>>>> 
>>>>>>>> Gideon,
>>>>>>>> 
>>>>>>>> In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>>>> 
>>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>>>>> 
>>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>>>> 
>>>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>>>> 
>>>>>>>>>>> 0 SNES Function norm 3.857327250267e-09 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>>> 1 SNES Function norm 2.479697868412e-12 
>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>>> 3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>>> 3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>>> 3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>>> 4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>>> 5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>>> 6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>>> 4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>>> 5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>>> 6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>>> 3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>>> 4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>>> 3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>>> 4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>>> 5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>>> 6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>>> 7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>>> 8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>>> 4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>>> 5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>>> 6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>>> 7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>>> 8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>>> 9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>>>> 10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>>>> 11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>>>> 12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>>>> 13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>>>> 14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>>>> 15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>>>> 16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>>>> 17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>>>> 18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>>>> 19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>>>> 20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>>>> 21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>>>> 22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>>>> 23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>>>> 24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>>>> 25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>>>> 26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>>>> 27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>>>> 28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>>>> 29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>>>> 30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>>>> 31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>>>> 32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>>>> 33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>>>> 34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>>>> 35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>>>> 36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>>>> 37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>>>> 38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>>>> 39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>>>> 40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>>>> 41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>>>> 42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>>>> 43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>>>> 44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>>>> 45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>>>> 46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>>>> 47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>>>> 48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>>>> 49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>>>> 50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>>>> 51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>>>> 52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>>>> 53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>>>> 54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>>>> 55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>>>> 56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>>>> 57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>>>> 58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>>>> 59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>>>> 60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>>>> 61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>>>> 62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>>>> 63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>>>> 64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>>>> 65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>>>> 66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>>>> 67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>>>> 68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>>>> 69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>>>> 70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>>>> 71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>>>> 72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>>>> 73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>>>> 74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>>>> 75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>>>> 76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>>>> 77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>>>> 78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>>>> 79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>>>> 80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>>>> 81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>>>> 82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>>>> 83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>>>> 84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>>>> 85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>>>> 86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>>>> 87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>>>> 88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>>>> 89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>>>> 90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>>>> 91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>>>> 92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>>>> 93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>>>> 94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>>>> 95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>>>> 96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>>>> 97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>>>> 98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>>>> 99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>>>> 
>>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>>>> 
>>>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>>>> 1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> then 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>>>> Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>>>>  type: seqaij
>>>>>>>>>>>>>>>>>>>>>  rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>>  package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>>>  total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>>>  total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>>>    MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>>>      SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>>>>      PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>>>>      ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>>>>      CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>>>>      CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>>>>      CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>>>>      CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>>>>      CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>>>>      RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>>>>        [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>      RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>>>>        [0]  143924 
>>>>>>>>>>>>>>>>>>>>>      RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>>>>        [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>      INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>      [0] 7 
>>>>>>>>>>>>>>>>>>>>>      INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>>>>>      INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>>>>        [0] 15991 
>>>>>>>>>>>>>>>>>>>>>      RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>      RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>>>>      RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>      (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>>>      INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>>>>      INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>>>>      INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>>>>      INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>>>>      INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>>>>      INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>>>>      INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>>>>      INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>>>>      INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>>>>      INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>>>>      INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>>>>      INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>>>>      INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>>>>      INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>      INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>      INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>      INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>      INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>>>>      INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>      INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>      INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>>>>      INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>>>>      INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>>>>      INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>>>      INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>>>      INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>>>      INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>>>      INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>>>      INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>>>  SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>>>  SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>>>  etc
>>>>>>>>>>>>>>>>>>>>>>>  SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>>>  SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>>  SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>>>  VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>>>  SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/42110a26/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Sep  8 20:38:13 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 Sep 2015 20:38:13 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <F30D7B58-9EFB-4BFF-AA4A-64605586B043@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>
	<EEA729D5-6242-4071-A8E1-9B0A752CAF24@mcs.anl.gov>
	<9DF2724E-BB51-4892-9E3D-DB6D7B286C39@gmail.com>
	<5B5109E8-6C37-4735-ACCE-54ED7C18E218@mcs.anl.gov>
	<F30D7B58-9EFB-4BFF-AA4A-64605586B043@gmail.com>
Message-ID: <AA097CC6-87D8-45FC-8D68-593FCF4FED3C@mcs.anl.gov>


> On Sep 8, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> The message was unclear.  I assumed that you meant if I went up to 3.6.1, this would be corrected.  What file am I editing and in which directory?

src/snes/interface/snesj.c function SNESComputeJacobianDefault


> 
> 
> -gideon
> 
>> On Sep 8, 2015, at 9:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  Did you add the line I told you to?
>> 
>> 
>>> On Sep 8, 2015, at 8:23 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> I just installed a fresh 3.6.1 pull, compiled against it, and I?m still having an error.
>>>  0 SNES Function norm 1.421454390131e-02 
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Argument out of range
>>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 21:21:11 2015
>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 485 in /opt/petsc/src/mat/impls/aij/seq/aij.c
>>> [0]PETSC ERROR: #2 MatSetValues() line 1173 in /opt/petsc/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 121 in /opt/petsc/src/snes/interface/snesj.c
>>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2232 in /opt/petsc/src/snes/interface/snes.c
>>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 231 in /opt/petsc/src/snes/impls/ls/ls.c
>>> [0]PETSC ERROR: #6 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Object is in wrong state
>>> [0]PETSC ERROR:  Vec is locked read only, argument # 1
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 21:21:11 2015
>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>> [0]PETSC ERROR: #7 VecGetArray() line 1646 in /opt/petsc/src/vec/vec/interface/rvector.c
>>> 
>>> 
>>> 
>>> 
>>> -gideon
>>> 
>>>> On Sep 8, 2015, at 8:56 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> This is our fault. You need to add a line 
>>>> 
>>>> ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);CHKERRQ(ierr);
>>>> 
>>>> after 
>>>> 
>>>> PetscFunctionBegin;
>>>> 
>>>> in SNESComputeJacobianDefault()
>>>> 
>>>> then do
>>>> 
>>>> make gnumake 
>>>> 
>>>> in that directory.
>>>> 
>>>> You really should update to PETSc 3.6 (http://www.mcs.anl.gov/petsc/download/index.html note it is best to obtain with git) It will make both your life and our lives easier and there is no downside.
>>>> 
>>>> Barry
>>>> 
>>>>> On Sep 8, 2015, at 6:24 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> When I use that flag, I get this error:
>>>>> 
>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>> [0]PETSC ERROR: Argument out of range
>>>>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
>>>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 19:24:08 2015
>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c
>>>>> [0]PETSC ERROR: #2 MatSetValues() line 1135 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in /opt/petsc-3.5.4/src/snes/interface/snesj.c
>>>>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>>>>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /opt/petsc-3.5.4/src/snes/impls/ls/ls.c
>>>>> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>>>>> 
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>>>>>> 
>>>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>>>>> 
>>>>>>>> Matt
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>>>>> 
>>>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>>>>> 
>>>>>>>>> Gideon,
>>>>>>>>> 
>>>>>>>>> In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>>>>> 
>>>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>>>>>> 
>>>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>>>>> 
>>>>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>>>>> 
>>>>>>>>>>>> 0 SNES Function norm 3.857327250267e-09 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>>>> 1 SNES Function norm 2.479697868412e-12 
>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>>>> 4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>>>> 5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>>>> 6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>>>> 4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>>>> 5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>>>> 6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>>>> 4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>>>> 4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>>>> 5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>>>> 6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>>>> 7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>>>> 8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>>>> 4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>>>> 5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>>>> 6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>>>> 7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>>>> 8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>>>> 9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>>>>> 10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>>>>> 11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>>>>> 12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>>>>> 13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>>>>> 14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>>>>> 15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>>>>> 16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>>>>> 17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>>>>> 18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>>>>> 19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>>>>> 20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>>>>> 21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>>>>> 22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>>>>> 23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>>>>> 24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>>>>> 25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>>>>> 26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>>>>> 27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>>>>> 28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>>>>> 29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>>>>> 30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>>>>> 31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>>>>> 32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>>>>> 33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>>>>> 34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>>>>> 35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>>>>> 36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>>>>> 37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>>>>> 38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>>>>> 39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>>>>> 40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>>>>> 41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>>>>> 42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>>>>> 43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>>>>> 44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>>>>> 45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>>>>> 46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>>>>> 47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>>>>> 48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>>>>> 49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>>>>> 50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>>>>> 51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>>>>> 52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>>>>> 53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>>>>> 54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>>>>> 55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>>>>> 56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>>>>> 57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>>>>> 58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>>>>> 59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>>>>> 60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>>>>> 61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>>>>> 62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>>>>> 63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>>>>> 64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>>>>> 65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>>>>> 66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>>>>> 67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>>>>> 68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>>>>> 69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>>>>> 70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>>>>> 71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>>>>> 72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>>>>> 73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>>>>> 74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>>>>> 75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>>>>> 76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>>>>> 77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>>>>> 78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>>>>> 79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>>>>> 80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>>>>> 81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>>>>> 82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>>>>> 83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>>>>> 84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>>>>> 85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>>>>> 86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>>>>> 87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>>>>> 88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>>>>> 89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>>>>> 90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>>>>> 91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>>>>> 92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>>>>> 93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>>>>> 94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>>>>> 95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>>>>> 96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>>>>> 97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>>>>> 98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>>>>> 99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>>>>> 1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> then 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>>>>> Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>>>>>  type: seqaij
>>>>>>>>>>>>>>>>>>>>>>  rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>>>  package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>>>>  total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>>>>  total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>>>>    MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>>>>      SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>>>>>      PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>>>>>      ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>>>>>      CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>>>>>      CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>>>>>      CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>>>>>      CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>>>>>      CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>>>>>      RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>>>>>        [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>>      RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>>>>>        [0]  143924 
>>>>>>>>>>>>>>>>>>>>>>      RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>>>>>        [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>>      INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>>      [0] 7 
>>>>>>>>>>>>>>>>>>>>>>      INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>>>>>>      INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>>>>>        [0] 15991 
>>>>>>>>>>>>>>>>>>>>>>      RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>>      RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>>>>>      RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>>      (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>>>>      INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>>>>>      INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>>>>      INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>>>>      INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>>>>      INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>>>>      INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>>>>      INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>>>>  SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>>>>  SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>>>>  etc
>>>>>>>>>>>>>>>>>>>>>>>>  SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>>>>  SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>>>  SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>>>>  VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>>>>  SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From gideon.simpson at gmail.com  Tue Sep  8 20:50:47 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 21:50:47 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <AA097CC6-87D8-45FC-8D68-593FCF4FED3C@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<15A8E9AD-4250-48E9-B43F-E3D9C9791E1D@gmail.com>
	<EEA729D5-6242-4071-A8E1-9B0A752CAF24@mcs.anl.gov>
	<9DF2724E-BB51-4892-9E3D-DB6D7B286C39@gmail.com>
	<5B5109E8-6C37-4735-ACCE-54ED7C18E218@mcs.anl.gov>
	<F30D7B58-9EFB-4BFF-AA4A-64605586B043@gmail.com>
	<AA097CC6-87D8-45FC-8D68-593FCF4FED3C@mcs.anl.gov>
Message-ID: <FE1869FE-6FEB-454A-95D0-23BFB363D03D@gmail.com>

Ok, that worked.
-gideon

> On Sep 8, 2015, at 9:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>> On Sep 8, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> The message was unclear.  I assumed that you meant if I went up to 3.6.1, this would be corrected.  What file am I editing and in which directory?
> 
> src/snes/interface/snesj.c function SNESComputeJacobianDefault
> 
> 
>> 
>> 
>> -gideon
>> 
>>> On Sep 8, 2015, at 9:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> Did you add the line I told you to?
>>> 
>>> 
>>>> On Sep 8, 2015, at 8:23 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> I just installed a fresh 3.6.1 pull, compiled against it, and I?m still having an error.
>>>> 0 SNES Function norm 1.421454390131e-02 
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: Argument out of range
>>>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
>>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 21:21:11 2015
>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 485 in /opt/petsc/src/mat/impls/aij/seq/aij.c
>>>> [0]PETSC ERROR: #2 MatSetValues() line 1173 in /opt/petsc/src/mat/interface/matrix.c
>>>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 121 in /opt/petsc/src/snes/interface/snesj.c
>>>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2232 in /opt/petsc/src/snes/interface/snes.c
>>>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 231 in /opt/petsc/src/snes/impls/ls/ls.c
>>>> [0]PETSC ERROR: #6 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: Object is in wrong state
>>>> [0]PETSC ERROR:  Vec is locked read only, argument # 1
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 21:21:11 2015
>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>> [0]PETSC ERROR: #7 VecGetArray() line 1646 in /opt/petsc/src/vec/vec/interface/rvector.c
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 8, 2015, at 8:56 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> This is our fault. You need to add a line 
>>>>> 
>>>>> ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);CHKERRQ(ierr);
>>>>> 
>>>>> after 
>>>>> 
>>>>> PetscFunctionBegin;
>>>>> 
>>>>> in SNESComputeJacobianDefault()
>>>>> 
>>>>> then do
>>>>> 
>>>>> make gnumake 
>>>>> 
>>>>> in that directory.
>>>>> 
>>>>> You really should update to PETSc 3.6 (http://www.mcs.anl.gov/petsc/download/index.html note it is best to obtain with git) It will make both your life and our lives easier and there is no downside.
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> On Sep 8, 2015, at 6:24 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> When I use that flag, I get this error:
>>>>>> 
>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>> [0]PETSC ERROR: Argument out of range
>>>>>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
>>>>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep  8 19:24:08 2015
>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c
>>>>>> [0]PETSC ERROR: #2 MatSetValues() line 1135 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in /opt/petsc-3.5.4/src/snes/interface/snesj.c
>>>>>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>>>>>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /opt/petsc-3.5.4/src/snes/impls/ls/ls.c
>>>>>> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>>>>>>> 
>>>>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>>>>>> 
>>>>>>>>> Matt
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>>>>>> 
>>>>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>>>>>> 
>>>>>>>>>> Gideon,
>>>>>>>>>> 
>>>>>>>>>> In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>>>>>> 
>>>>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>>>>>>> 
>>>>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 0 SNES Function norm 3.857327250267e-09 
>>>>>>>>>>>>>  0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>>>>>  2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>>>>> 1 SNES Function norm 2.479697868412e-12 
>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>>>>> 4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>>>>> 5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>>>>> 6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>>>>> 4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>>>>> 5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>>>>> 6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>>>>> 4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>>>>> 4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>>>>> 5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>>>>> 6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>>>>> 7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>>>>> 8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>>>>> 4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>>>>> 5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>>>>> 6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>>>>> 7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>>>>> 8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>>>>> 9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>>>>>> 10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>>>>>> 11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>>>>>> 12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>>>>>> 13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>>>>>> 14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>>>>>> 15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>>>>>> 16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>>>>>> 17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>>>>>> 18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>>>>>> 19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>>>>>> 20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>>>>>> 21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>>>>>> 22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>>>>>> 23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>>>>>> 24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>>>>>> 25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>>>>>> 26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>>>>>> 27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>>>>>> 28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>>>>>> 29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>>>>>> 30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>>>>>> 31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>>>>>> 32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>>>>>> 33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>>>>>> 34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>>>>>> 35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>>>>>> 36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>>>>>> 37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>>>>>> 38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>>>>>> 39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>>>>>> 40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>>>>>> 41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>>>>>> 42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>>>>>> 43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>>>>>> 44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>>>>>> 45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>>>>>> 46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>>>>>> 47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>>>>>> 48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>>>>>> 49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>>>>>> 50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>>>>>> 51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>>>>>> 52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>>>>>> 53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>>>>>> 54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>>>>>> 55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>>>>>> 56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>>>>>> 57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>>>>>> 58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>>>>>> 59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>>>>>> 60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>>>>>> 61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>>>>>> 62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>>>>>> 63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>>>>>> 64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>>>>>> 65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>>>>>> 66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>>>>>> 67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>>>>>> 68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>>>>>> 69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>>>>>> 70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>>>>>> 71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>>>>>> 72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>>>>>> 73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>>>>>> 74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>>>>>> 75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>>>>>> 76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>>>>>> 77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>>>>>> 78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>>>>>> 79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>>>>>> 80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>>>>>> 81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>>>>>> 82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>>>>>> 83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>>>>>> 84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>>>>>> 85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>>>>>> 86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>>>>>> 87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>>>>>> 88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>>>>>> 89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>>>>>> 90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>>>>>> 91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>>>>>> 92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>>>>>> 93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>>>>>> 94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>>>>>> 95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>>>>>> 96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>>>>>> 97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>>>>>> 98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>>>>>> 99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>>>>>> 1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> then 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>>>>>> Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>>>> package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>>>>> total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>>>>>   MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>>>>>     SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>>>>>>     PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>>>>>>     ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>>>>>>     CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>>>>>>     CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>>>>>>     CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>>>>>>     CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>>>>>>     CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>>>>>>     RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>>>>>>       [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>>>     RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>>>>>>       [0]  143924 
>>>>>>>>>>>>>>>>>>>>>>>     RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>>>>>>       [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>>>     INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>>>     [0] 7 
>>>>>>>>>>>>>>>>>>>>>>>     INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>>>       [0] 7 
>>>>>>>>>>>>>>>>>>>>>>>     INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>>>>>>       [0] 15991 
>>>>>>>>>>>>>>>>>>>>>>>     RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>>>     RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>>>>>>     RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>>>     (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>>>>>     INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>>>>> SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>>>>> SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>>>>> etc
>>>>>>>>>>>>>>>>>>>>>>>>> SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>>>> SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>>>>> VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>>>>> SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/6f1e26b7/attachment-0001.html>

From gideon.simpson at gmail.com  Tue Sep  8 20:51:53 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 21:51:53 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
Message-ID: <7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>

Going back to this point, a case that previously worked, is now giving me problems that I don?t understand:

  0 SNES Function norm 1.421454390131e-02 
    0 KSP preconditioned resid norm 7.635339584553e-03 true resid norm 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.838956718317e-07 true resid norm 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
    2 KSP preconditioned resid norm 5.889748199467e-12 true resid norm 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
  1 SNES Function norm 4.476139939604e-06 
    0 KSP preconditioned resid norm 4.897102387844e-05 true resid norm 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.133649519960e-09 true resid norm 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
    2 KSP preconditioned resid norm 2.223328042462e-14 true resid norm 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
  2 SNES Function norm 5.447761863330e-09 
    0 KSP preconditioned resid norm 9.703890346186e-09 true resid norm 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.982949185218e-13 true resid norm 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
    2 KSP preconditioned resid norm 7.045835741315e-18 true resid norm 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT iterations 2

-gideon

> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
> 
>  Barry
> 
>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>> 
>> 
>> -gideon
>> 
>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>> 
>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>> PETSc refines the DA and passes in the new one something goes wrong?
>>> 
>>>   Matt
>>> 
>>> -gideon
>>> 
>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>> 
>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>> 
>>>> Gideon,
>>>> 
>>>>  In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>> 
>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>> 
>>>> 
>>>> Barry
>>>> 
>>>>> 
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>> 
>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>> 
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>> 
>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>> 
>>>>>>>   0 SNES Function norm 3.857327250267e-09 
>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>   1 SNES Function norm 2.479697868412e-12 
>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>> 
>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>> 
>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>> 
>>>>>>>>>  0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>  1 SNES Function norm 2.177599365111e-12 
>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>> 
>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>> 
>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>> 
>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>> 
>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Hmm,
>>>>>>>>>>>> 
>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>> 
>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>> 
>>>>>>>>>>>> then 
>>>>>>>>>>>> 
>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>> 
>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>    type: seqaij
>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>      MUMPS run parameters:
>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>        PAR (host participation):            1 
>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>          [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>          [0]  143924 
>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>          [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>          [0] 15991 
>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>    SNESSetDM()
>>>>>>>>>>>>>>>>>>>    etc
>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/21e1ec94/attachment-0001.html>

From knepley at gmail.com  Tue Sep  8 20:55:18 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Sep 2015 20:55:18 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
Message-ID: <CAMYG4G=XKBXq_=+YUsqer-CR7g=1dkaTWQVmLzWVgJKZdKgzCQ@mail.gmail.com>

On Tue, Sep 8, 2015 at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> I went through my code and nothing jumps out at me as clearly wrong.  Is
> there a way to dump to disk vectors and matrices that would be useful for
> the comparison?  I imagine I?d want the jacobian that I am specifying, but
> what would I want to compare that against in terms of using
> -snes_mf_operator?
>

What you really want is to be able to use -snes_test, but that requires a
full Jacobian. You can get one
using -snes_fd, but its dense. Our default is usually -snes_fd_color, but
we have no coloring for DMComposite,
so that is not done. This is the advantage of using a combined formulation
on a single grid.

  Thanks,

     Matt


> -gideon
>
> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com>
>  wrote:
>
>> Just to clarify, again, during the linear solve phase, even though the PC
>> residual norm was small (getting to 10^{-10} in 2-3 iterations) that the
>> true residual norm could be comparatively large signified there is
>> something wrong.  The one thing i was curious about is why it seems that
>> there is only an issue during grid sequencing.  I?m not seeing that kind of
>> behavior on the coarsest mesh.
>>
>
> That definitely sounds like a bug in the Jacobian evaluation. Maybe you
> assume a certain mesh size, and when
> PETSc refines the DA and passes in the new one something goes wrong?
>
>    Matt
>
>
>> -gideon
>>
>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> Alright, I?ll take another look at what has been implemented in my
>> Jacobian file.  One thing I?m a bit unclear about is do I need these
>>  -snes_mf and snes_mf_operator flags if I have, coded in,
>>
>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>
>>
>> Gideon,
>>
>>   In your case the DM creates that Jacobian matrix object and
>> form_function_jacobian's job is just to fill it up with the correct values
>> for the current solution value.   If you use -snes_mf_operator then SNES
>> uses a matrix-free multiple for the first matrix and your Jacobian to build
>> the preconditioner. With -snes_mf it uses a matrix-free multiple for the
>> first and second matrix and ignores what you compute. Since it is matrix
>> free in both it cannot build a preconditioner hence the -pc_type none
>>
>>  Using -snes_mf and -snes_mf_operator are a way of testing if the
>> Jacobian you provide is correct
>>
>>
>>  Barry
>>
>>
>>
>> -gideon
>>
>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> If the Jacobian you provide is correct then using -snes_mf with a PC type
>> of LU should give two iterations (or at most three) to get very good
>> (10^-12) convergence of the linear system. Here you are not getting that
>> after a few SNES iterations. So I am strongly leaning to something wrong
>> with the Jacobian you provide.  -snes_type test unfortunately doesn't catch
>> all problems.
>>
>> You can try -snes_mf -pc_type none and see how the linear solver
>> converges.
>>
>>
>> Barry
>>
>>
>>
>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> Off the top of my head, I can?t rule out a null space in this problem. I
>> thought I did a pretty good job of coding the Jacobian by hand.  And before
>> I made that change to the Jacobian file, changing J for Jpre, if I ran with
>>  -snes_check_jacobian, it appeared to be satisfactory.
>>
>> When I run with -ksp_type fgmres, I get
>>
>>    0 SNES Function norm 3.857327250267e-09
>>      0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm
>> 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm
>> 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>      2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm
>> 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>    1 SNES Function norm 2.479697868412e-12
>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>  0 SNES Function norm 5.066222213176e+03
>>    0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm
>> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm
>> 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>  1 SNES Function norm 8.484696862318e+02
>>    0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm
>> 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm
>> 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>    2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm
>> 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>  2 SNES Function norm 6.551139809777e+02
>>    0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm
>> 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm
>> 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>    2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm
>> 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>  3 SNES Function norm 5.771483337231e+02
>>    0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm
>> 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm
>> 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>    2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm
>> 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>  4 SNES Function norm 5.237540245670e+02
>>    0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm
>> 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm
>> 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>    2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm
>> 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>  5 SNES Function norm 4.753058321803e+02
>>    0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm
>> 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm
>> 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>    2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm
>> 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>  6 SNES Function norm 4.221482124411e+02
>>    0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm
>> 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm
>> 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>    2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm
>> 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>  7 SNES Function norm 3.803620028964e+02
>>    0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm
>> 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm
>> 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>    2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm
>> 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>  8 SNES Function norm 3.769519577570e+02
>>    0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm
>> 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm
>> 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>    2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm
>> 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>    3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm
>> 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>  9 SNES Function norm 3.766389203406e+02
>>    0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm
>> 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm
>> 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>    2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm
>> 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>> 10 SNES Function norm 3.751822489648e+02
>>    0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm
>> 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm
>> 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>    2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm
>> 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>> 11 SNES Function norm 3.737189963925e+02
>>    0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm
>> 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm
>> 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>    2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm
>> 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>    3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm
>> 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>> 12 SNES Function norm 3.737110331594e+02
>>    0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm
>> 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm
>> 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>    2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm
>> 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>> 13 SNES Function norm 3.684175513154e+02
>>    0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm
>> 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm
>> 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>    2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm
>> 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>> 14 SNES Function norm 3.509930783933e+02
>>    0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm
>> 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm
>> 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>    2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm
>> 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>> 15 SNES Function norm 3.162121096565e+02
>>    0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm
>> 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm
>> 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>    2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm
>> 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>> 16 SNES Function norm 2.890165052818e+02
>>    0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm
>> 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm
>> 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>    2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm
>> 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>> 17 SNES Function norm 2.816929421445e+02
>>    0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm
>> 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm
>> 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>    2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm
>> 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>    3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm
>> 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>> 18 SNES Function norm 2.814407559738e+02
>>    0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm
>> 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm
>> 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>    2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm
>> 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>> 19 SNES Function norm 2.810261821835e+02
>>    0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm
>> 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm
>> 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>    2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm
>> 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>> 20 SNES Function norm 2.755723672077e+02
>>    0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm
>> 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm
>> 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>    2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm
>> 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>    3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm
>> 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>    4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm
>> 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>    5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm
>> 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>    6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm
>> 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>> 21 SNES Function norm 2.755720389981e+02
>>    0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm
>> 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm
>> 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>    2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm
>> 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>    3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm
>> 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>    4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm
>> 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>    5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm
>> 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>    6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm
>> 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>> 22 SNES Function norm 2.755716464881e+02
>>    0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm
>> 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm
>> 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>    2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm
>> 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>    3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm
>> 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>    4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm
>> 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>> 23 SNES Function norm 2.755713163168e+02
>>    0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm
>> 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm
>> 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>    2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm
>> 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>    3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm
>> 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>    4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm
>> 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>    5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm
>> 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>    6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm
>> 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>    7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm
>> 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>    8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm
>> 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>> 24 SNES Function norm 2.755712363532e+02
>>    0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm
>> 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm
>> 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>    2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm
>> 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>    3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm
>> 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>    4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm
>> 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>    5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm
>> 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>    6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm
>> 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>    7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm
>> 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>    8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm
>> 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>    9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm
>> 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>   10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm
>> 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>   11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm
>> 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>   12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm
>> 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>   13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm
>> 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>   14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm
>> 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>   15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm
>> 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>   16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm
>> 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>   17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm
>> 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>   18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm
>> 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>   19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm
>> 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>   20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm
>> 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>   21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm
>> 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>   22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm
>> 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>   23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm
>> 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>   24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm
>> 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>   25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm
>> 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>   26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm
>> 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>   27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm
>> 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>   28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm
>> 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>   29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm
>> 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>   30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm
>> 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>   31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm
>> 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>   32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm
>> 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>   33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm
>> 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>   34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm
>> 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>   35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm
>> 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>   36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm
>> 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>   37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm
>> 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>   38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm
>> 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>   39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm
>> 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>   40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm
>> 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>   41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm
>> 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>   42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm
>> 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>   43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm
>> 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>   44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm
>> 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>   45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm
>> 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>   46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm
>> 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>   47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm
>> 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>   48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm
>> 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>   49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm
>> 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>   50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm
>> 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>   51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm
>> 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>   52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm
>> 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>   53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm
>> 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>   54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm
>> 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>   55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm
>> 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>   56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm
>> 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>   57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm
>> 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>   58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm
>> 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>   59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm
>> 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>   60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm
>> 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>   61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm
>> 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>   62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm
>> 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>   63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm
>> 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>   64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm
>> 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>   65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm
>> 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>   66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm
>> 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>   67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm
>> 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>   68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm
>> 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>   69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm
>> 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>   70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm
>> 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>   71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm
>> 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>   72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm
>> 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>   73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm
>> 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>   74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm
>> 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>   75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm
>> 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>   76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm
>> 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>   77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm
>> 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>   78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm
>> 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>   79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm
>> 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>   80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm
>> 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>   81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm
>> 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>   82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm
>> 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>   83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm
>> 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>   84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm
>> 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>   85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm
>> 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>   86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm
>> 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>   87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm
>> 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>   88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm
>> 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>   89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm
>> 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>   90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm
>> 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>   91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm
>> 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>   92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm
>> 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>   93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm
>> 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>   94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm
>> 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>   95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm
>> 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>   96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm
>> 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>   97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm
>> 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>   98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm
>> 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>   99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm
>> 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>  100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm
>> 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>  101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm
>> 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>  102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm
>> 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>  103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm
>> 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>  104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm
>> 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>  105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm
>> 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>  106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm
>> 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>  107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm
>> 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>  108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm
>> 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>  109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm
>> 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>  110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm
>> 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>  111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm
>> 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>  112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm
>> 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>  113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm
>> 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>  114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm
>> 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>  115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm
>> 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>  116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm
>> 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>  117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm
>> 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>  118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm
>> 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>  119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm
>> 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>  120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm
>> 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>  121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm
>> 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>  122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm
>> 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>  123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm
>> 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>  124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm
>> 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>  125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm
>> 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>  126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm
>> 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>  127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm
>> 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>  128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm
>> 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>  129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm
>> 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>  130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm
>> 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>  131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm
>> 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>  132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm
>> 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>  133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm
>> 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>  134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm
>> 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>  135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm
>> 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>  136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm
>> 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>  137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm
>> 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>  138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm
>> 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>  139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm
>> 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>  140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm
>> 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>  141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm
>> 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>  142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm
>> 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>  143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm
>> 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>  144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm
>> 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>  145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm
>> 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>  146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm
>> 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>  147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm
>> 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>  148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm
>> 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>  149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm
>> 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>  150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm
>> 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>  151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm
>> 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>  152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm
>> 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>  153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm
>> 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>  154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm
>> 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>  155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm
>> 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>  156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm
>> 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>  157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm
>> 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>  158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm
>> 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>  159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm
>> 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>  160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm
>> 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>  161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm
>> 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>  162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm
>> 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>  163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm
>> 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>  164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm
>> 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>  165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm
>> 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>  166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm
>> 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>  167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm
>> 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>  168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm
>> 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>  169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm
>> 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>  170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm
>> 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>  171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm
>> 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>  172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm
>> 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>  173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm
>> 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>  174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm
>> 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>  175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm
>> 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>  176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm
>> 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>  177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm
>> 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>  178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm
>> 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>  179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm
>> 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>  180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm
>> 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>  181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm
>> 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>  182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm
>> 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>  183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm
>> 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>  184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm
>> 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>  185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm
>> 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>  186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm
>> 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>  187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm
>> 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>  188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm
>> 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>  189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm
>> 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>  190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm
>> 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>  191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm
>> 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>  192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm
>> 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>  193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm
>> 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>  194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm
>> 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>  195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm
>> 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>  196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm
>> 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>  197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm
>> 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>  198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm
>> 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>  199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm
>> 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>  200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm
>> 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>  201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm
>> 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>  202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm
>> 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>  203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm
>> 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>  204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm
>> 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>  205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm
>> 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>  206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm
>> 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>  207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm
>> 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>  208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm
>> 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>  209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm
>> 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>  210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm
>> 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations
>> 24
>>
>> -gideon
>>
>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> Is there any chance your Jacobian may have a null space? (For example
>> with Neumann boundary conditions there is often a nullspace)
>>
>> If not run with with -ksp_type fgmres and send the same output.
>>
>> Something is screwy with the linear system convergence, could be several
>> things including either a null space problem or that your Jacobian that you
>> provide is terribly wrong.
>>
>> Barry
>>
>>
>> 21 SNES Function norm 2.987064584431e+02
>>   0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm
>> 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm
>> 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>   2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm
>> 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>
>>
>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> Got it, fixing that, and returning to the original question, this is what
>> I now get, when I use those two flags:
>>
>>   0 SNES Function norm 1.132185384796e-08
>>     0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm
>> 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>     1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm
>> 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>   1 SNES Function norm 2.177599365111e-12
>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>> 0 SNES Function norm 5.066222213176e+03
>>   0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm
>> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm
>> 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>> 1 SNES Function norm 8.482593852817e+02
>>   0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm
>> 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm
>> 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>   2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm
>> 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>> 2 SNES Function norm 6.543140468549e+02
>>   0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm
>> 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm
>> 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>   2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm
>> 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>> 3 SNES Function norm 5.766430557220e+02
>>   0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm
>> 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm
>> 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>   2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm
>> 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>> 4 SNES Function norm 5.235211958260e+02
>>   0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm
>> 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm
>> 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>   2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm
>> 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>> 5 SNES Function norm 4.752913229649e+02
>>   0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm
>> 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm
>> 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>   2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm
>> 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>> 6 SNES Function norm 4.220255380391e+02
>>   0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm
>> 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm
>> 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>   2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm
>> 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>> 7 SNES Function norm 3.805408907074e+02
>>   0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm
>> 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm
>> 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>> 8 SNES Function norm 3.764619752339e+02
>>   0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm
>> 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm
>> 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>   2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm
>> 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>> 9 SNES Function norm 3.761182227091e+02
>>   0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm
>> 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm
>> 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>   2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm
>> 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>> 10 SNES Function norm 3.740017190063e+02
>>   0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm
>> 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm
>> 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>   2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm
>> 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>> 11 SNES Function norm 3.725903477238e+02
>>   0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm
>> 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm
>> 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>   2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm
>> 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>> 12 SNES Function norm 3.716162097231e+02
>>   0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm
>> 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm
>> 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>   2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm
>> 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>> 13 SNES Function norm 3.674168632847e+02
>>   0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm
>> 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm
>> 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>   2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm
>> 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>> 14 SNES Function norm 3.532395445266e+02
>>   0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm
>> 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm
>> 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>   2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm
>> 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>> 15 SNES Function norm 3.182438872366e+02
>>   0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm
>> 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm
>> 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>   2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm
>> 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>> 16 SNES Function norm 3.091759892779e+02
>>   0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm
>> 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm
>> 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>   2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm
>> 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>> 17 SNES Function norm 2.987839504359e+02
>>   0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm
>> 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm
>> 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>   2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm
>> 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>> 18 SNES Function norm 2.987073622777e+02
>>   0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm
>> 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm
>> 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>   2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm
>> 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>> 19 SNES Function norm 2.987067936734e+02
>>   0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm
>> 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm
>> 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>   2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm
>> 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>> 20 SNES Function norm 2.987067502910e+02
>>   0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm
>> 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm
>> 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>   2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm
>> 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>> 21 SNES Function norm 2.987064584431e+02
>>   0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm
>> 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm
>> 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>   2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm
>> 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>> 22 SNES Function norm 2.987064525262e+02
>>   0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm
>> 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm
>> 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>   2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm
>> 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>> 23 SNES Function norm 2.987064121622e+02
>>   0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm
>> 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm
>> 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>   2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm
>> 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>> 24 SNES Function norm 2.987063973426e+02
>>   0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm
>> 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm
>> 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>   2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm
>> 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>> 25 SNES Function norm 2.987063920553e+02
>>   0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm
>> 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm
>> 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>   2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm
>> 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>> 26 SNES Function norm 2.987063919786e+02
>>   0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm
>> 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm
>> 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>   2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm
>> 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>> 27 SNES Function norm 2.987063393145e+02
>>   0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm
>> 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm
>> 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>   2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm
>> 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>> 28 SNES Function norm 2.987063253277e+02
>>   0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm
>> 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm
>> 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>   2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm
>> 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>> 29 SNES Function norm 2.987063197153e+02
>>   0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm
>> 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm
>> 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>   2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm
>> 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>> 30 SNES Function norm 2.987063193089e+02
>>   0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm
>> 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm
>> 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>   2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm
>> 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>> 31 SNES Function norm 2.987063192061e+02
>>   0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm
>> 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm
>> 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>   2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm
>> 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>> 32 SNES Function norm 2.987063094492e+02
>>   0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm
>> 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm
>> 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>   2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm
>> 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>> 33 SNES Function norm 2.987063089942e+02
>>   0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm
>> 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm
>> 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>   2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm
>> 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>> 34 SNES Function norm 2.987062660109e+02
>>   0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm
>> 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm
>> 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>   2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm
>> 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>> 35 SNES Function norm 2.987062107900e+02
>>   0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm
>> 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm
>> 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>   2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm
>> 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>> 36 SNES Function norm 2.987062055224e+02
>>   0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm
>> 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm
>> 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>   2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm
>> 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>> 37 SNES Function norm 2.987061774798e+02
>>   0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm
>> 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm
>> 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>   2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm
>> 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>> 38 SNES Function norm 2.987061715400e+02
>>   0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm
>> 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm
>> 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>   2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm
>> 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>> 39 SNES Function norm 2.987061699634e+02
>>   0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm
>> 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm
>> 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>   2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm
>> 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>> 40 SNES Function norm 2.987061630064e+02
>>   0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm
>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm
>> 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>> 41 SNES Function norm 2.987061630064e+02
>>   0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm
>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm
>> 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>> 42 SNES Function norm 2.987061630064e+02
>>   0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm
>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm
>> 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>> 43 SNES Function norm 2.987061630064e+02
>>   0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm
>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm
>> 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>> 44 SNES Function norm 2.987061630064e+02
>>   0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm
>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm
>> 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>> 45 SNES Function norm 2.987061630064e+02
>>   0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm
>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm
>> 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>> 46 SNES Function norm 2.987061630064e+02
>>   0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm
>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm
>> 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>> 47 SNES Function norm 2.987061630064e+02
>>   0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm
>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm
>> 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>
>>
>> -gideon
>>
>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> This indicates that somewhere in your ComputeJacobian you are setting
>> matrix entries with the first Mat argument when you should always set them
>> with the second matrix argument. For example if you have
>>
>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>
>> you should call all the MatSetValues() with jpre, no J. Then at the end
>> of the function you should call MatAssemblyBegin/End() on jpre then on J if
>> J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>
>> This is a minor glitch we'll get past.
>>
>> Barry
>>
>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> By the way, I tried using a different petsc installation, and now, rather
>> than the segmentation fault, I get the following error:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: No support for this operation for this object type
>> [0]PETSC ERROR: Mat type mffd
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
>> trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named
>> gs_air by gideon Mon Sep  7 21:32:18 2015
>> [0]PETSC ERROR: Configure options --download-mpich=yes
>> --download-suitesparse=yes --download-superlu=yes
>> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes
>> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes
>> --download-parmetis=yes --download-scalapack=yes
>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in
>> /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>
>> -gideon
>>
>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> Hmm,
>>
>> Ok you can try running it directly in the debugger since it is one
>> process, type
>>
>> gdb ./blowup_batch_refine
>>
>> then
>>
>> when the debugger comes up (if it does not cut and paste all output and
>> send it)
>>
>> run -on_error_abort -snes_mf_operator  and any other options you normally
>> use
>>
>>
>> Barry
>>
>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> Running with that flag gives me this:
>>
>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid
>> 16111 on gs_air
>> Unable to start debugger: No such file or directory
>>
>>
>>
>> -gideon
>>
>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> This should not happen. Run with a debug version of PETSc installed and
>> the option -start_in_debugger noxterm  Once the debugger starts up type
>> cont and when it crashes type where or bt  Send all output
>>
>>
>>
>> Barry
>>
>>
>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> I?m getting an error with -snes_mf_operator,
>>
>> 0 SNES Function norm 1.421454390131e-02
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
>> X to find memory corruption errors
>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
>> and run
>> [0]PETSC ERROR: to get more information on the crash.
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Signal received
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
>> trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by
>> gideon Mon Sep  7 21:08:19 2015
>> [0]PETSC ERROR: Configure options --prefix=/opt/local
>> --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries
>> --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0
>> --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate
>> --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local
>> --with-superlu-dir=/opt/local --with-metis-dir=/opt/local
>> --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local
>> --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local
>> CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp
>> FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp
>> F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os
>> FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names"
>> CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os
>> FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports
>> --with-mpiexec=mpiexec-mpich-mp
>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>
>> -gideon
>>
>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> My guess is the Jacobian is not correct (or correct "enough"), hence
>> PETSc SNES is generating a poor descent direction. You can try
>> -snes_mf_operator -ksp_monitor_true residual as additional arguments.
>> What happens?
>>
>> Barry
>>
>>
>>
>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> No problem Matt, I don?t think we had previously discussed that output.
>> Here is a case where things fail.
>>
>> 0 SNES Function norm 4.027481756921e-09
>> 1 SNES Function norm 1.760477878365e-12
>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>> 0 SNES Function norm 5.066222213176e+03
>> 1 SNES Function norm 8.484697184230e+02
>> 2 SNES Function norm 6.549559723294e+02
>> 3 SNES Function norm 5.770723278153e+02
>> 4 SNES Function norm 5.237702240594e+02
>> 5 SNES Function norm 4.753909019848e+02
>> 6 SNES Function norm 4.221784590755e+02
>> 7 SNES Function norm 3.806525080483e+02
>> 8 SNES Function norm 3.762054656019e+02
>> 9 SNES Function norm 3.758975226873e+02
>> 10 SNES Function norm 3.757032042706e+02
>> 11 SNES Function norm 3.728798164234e+02
>> 12 SNES Function norm 3.723078741075e+02
>> 13 SNES Function norm 3.721848059825e+02
>> 14 SNES Function norm 3.720227575629e+02
>> 15 SNES Function norm 3.720051998555e+02
>> 16 SNES Function norm 3.718945430587e+02
>> 17 SNES Function norm 3.700412694044e+02
>> 18 SNES Function norm 3.351964889461e+02
>> 19 SNES Function norm 3.096016086233e+02
>> 20 SNES Function norm 3.008410789787e+02
>> 21 SNES Function norm 2.752316716557e+02
>> 22 SNES Function norm 2.707658474165e+02
>> 23 SNES Function norm 2.698436736049e+02
>> 24 SNES Function norm 2.618233857172e+02
>> 25 SNES Function norm 2.600121920634e+02
>> 26 SNES Function norm 2.585046423168e+02
>> 27 SNES Function norm 2.568551090220e+02
>> 28 SNES Function norm 2.556404537064e+02
>> 29 SNES Function norm 2.536353523683e+02
>> 30 SNES Function norm 2.533596070171e+02
>> 31 SNES Function norm 2.532324379596e+02
>> 32 SNES Function norm 2.531842335211e+02
>> 33 SNES Function norm 2.531684527520e+02
>> 34 SNES Function norm 2.531637604618e+02
>> 35 SNES Function norm 2.531624767821e+02
>> 36 SNES Function norm 2.531621359093e+02
>> 37 SNES Function norm 2.531620504925e+02
>> 38 SNES Function norm 2.531620350055e+02
>> 39 SNES Function norm 2.531620310522e+02
>> 40 SNES Function norm 2.531620300471e+02
>> 41 SNES Function norm 2.531620298084e+02
>> 42 SNES Function norm 2.531620297478e+02
>> 43 SNES Function norm 2.531620297324e+02
>> 44 SNES Function norm 2.531620297303e+02
>> 45 SNES Function norm 2.531620297302e+02
>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>> 0 SNES Function norm 9.636339304380e+03
>> 1 SNES Function norm 8.997731184634e+03
>> 2 SNES Function norm 8.120498349232e+03
>> 3 SNES Function norm 7.322379894820e+03
>> 4 SNES Function norm 6.599581599149e+03
>> 5 SNES Function norm 6.374872854688e+03
>> 6 SNES Function norm 6.372518007653e+03
>> 7 SNES Function norm 6.073996314301e+03
>> 8 SNES Function norm 5.635965277054e+03
>> 9 SNES Function norm 5.155389064046e+03
>> 10 SNES Function norm 5.080567902638e+03
>> 11 SNES Function norm 5.058878643969e+03
>> 12 SNES Function norm 5.058835649793e+03
>> 13 SNES Function norm 5.058491285707e+03
>> 14 SNES Function norm 5.057452865337e+03
>> 15 SNES Function norm 5.057226140688e+03
>> 16 SNES Function norm 5.056651272898e+03
>> 17 SNES Function norm 5.056575190057e+03
>> 18 SNES Function norm 5.056574632598e+03
>> 19 SNES Function norm 5.056574520229e+03
>> 20 SNES Function norm 5.056574492569e+03
>> 21 SNES Function norm 5.056574485124e+03
>> 22 SNES Function norm 5.056574483029e+03
>> 23 SNES Function norm 5.056574482427e+03
>> 24 SNES Function norm 5.056574482302e+03
>> 25 SNES Function norm 5.056574482287e+03
>> 26 SNES Function norm 5.056574482282e+03
>> 27 SNES Function norm 5.056574482281e+03
>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>> SNES Object: 1 MPI processes
>> type: newtonls
>> maximum iterations=50, maximum function evaluations=10000
>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>> total number of linear solver iterations=28
>> total number of function evaluations=323
>> total number of grid sequence refinements=2
>> SNESLineSearch Object:   1 MPI processes
>> type: bt
>> interpolation: cubic
>> alpha=1.000000e-04
>> maxstep=1.000000e+08, minlambda=1.000000e-12
>> tolerances: relative=1.000000e-08, absolute=1.000000e-15,
>> lambda=1.000000e-08
>> maximum iterations=40
>> KSP Object:   1 MPI processes
>> type: gmres
>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> GMRES: happy breakdown tolerance 1e-30
>> maximum iterations=10000, initial guess is zero
>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> left preconditioning
>> using PRECONDITIONED norm type for convergence test
>> PC Object:   1 MPI processes
>> type: lu
>> LU: out-of-place factorization
>> tolerance for zero pivot 2.22045e-14
>> matrix ordering: nd
>> factor fill ratio given 0, needed 0
>> Factored matrix follows:
>>   Mat Object:           1 MPI processes
>>     type: seqaij
>>     rows=15991, cols=15991
>>     package used to perform factorization: mumps
>>     total: nonzeros=255801, allocated nonzeros=255801
>>     total number of mallocs used during MatSetValues calls =0
>>       MUMPS run parameters:
>>         SYM (matrix type):                   0
>>         PAR (host participation):            1
>>         ICNTL(1) (output for error):         6
>>         ICNTL(2) (output of diagnostic msg): 0
>>         ICNTL(3) (output for global info):   0
>>         ICNTL(4) (level of printing):        0
>>         ICNTL(5) (input mat struct):         0
>>         ICNTL(6) (matrix prescaling):        7
>>         ICNTL(7) (sequentia matrix ordering):6
>>         ICNTL(8) (scalling strategy):        77
>>         ICNTL(10) (max num of refinements):  0
>>         ICNTL(11) (error analysis):          0
>>         ICNTL(12) (efficiency control):                         1
>>         ICNTL(13) (efficiency control):                         0
>>         ICNTL(14) (percentage of estimated workspace increase): 20
>>         ICNTL(18) (input mat struct):                           0
>>         ICNTL(19) (Shur complement info):                       0
>>         ICNTL(20) (rhs sparse pattern):                         0
>>         ICNTL(21) (somumpstion struct):                            0
>>         ICNTL(22) (in-core/out-of-core facility):               0
>>         ICNTL(23) (max size of memory can be allocated locally):0
>>         ICNTL(24) (detection of null pivot rows):               0
>>         ICNTL(25) (computation of a null space basis):          0
>>         ICNTL(26) (Schur options for rhs or solution):          0
>>         ICNTL(27) (experimental parameter):                     -8
>>         ICNTL(28) (use parallel or sequential ordering):        1
>>         ICNTL(29) (parallel ordering):                          0
>>         ICNTL(30) (user-specified set of entries in inv(A)):    0
>>         ICNTL(31) (factors is discarded in the solve phase):    0
>>         ICNTL(33) (compute determinant):                        0
>>         CNTL(1) (relative pivoting threshold):      0.01
>>         CNTL(2) (stopping criterion of refinement): 1.49012e-08
>>         CNTL(3) (absomumpste pivoting threshold):      0
>>         CNTL(4) (vamumpse of static pivoting):         -1
>>         CNTL(5) (fixation for null pivots):         0
>>         RINFO(1) (local estimated flops for the elimination after
>> analysis):
>>           [0] 1.95838e+06
>>         RINFO(2) (local estimated flops for the assembly after
>> factorization):
>>           [0]  143924
>>         RINFO(3) (local estimated flops for the elimination after
>> factorization):
>>           [0]  1.95943e+06
>>         INFO(15) (estimated size of (in MB) MUMPS internal data for
>> running numerical factorization):
>>         [0] 7
>>         INFO(16) (size of (in MB) MUMPS internal data used during
>> numerical factorization):
>>           [0] 7
>>         INFO(23) (num of pivots eliminated on this processor after
>> factorization):
>>           [0] 15991
>>         RINFOG(1) (global estimated flops for the elimination after
>> analysis): 1.95838e+06
>>         RINFOG(2) (global estimated flops for the assembly after
>> factorization): 143924
>>         RINFOG(3) (global estimated flops for the elimination after
>> factorization): 1.95943e+06
>>         (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>         INFOG(3) (estimated real workspace for factors on all processors
>> after analysis): 255801
>>         INFOG(4) (estimated integer workspace for factors on all
>> processors after analysis): 127874
>>         INFOG(5) (estimated maximum front size in the complete tree): 11
>>         INFOG(6) (number of nodes in the complete tree): 3996
>>         INFOG(7) (ordering option effectively use after analysis): 6
>>         INFOG(8) (structural symmetry in percent of the permuted matrix
>> after analysis): 86
>>         INFOG(9) (total real/complex workspace to store the matrix
>> factors after factorization): 255865
>>         INFOG(10) (total integer space store the matrix factors after
>> factorization): 127890
>>         INFOG(11) (order of largest frontal matrix after factorization):
>> 11
>>         INFOG(12) (number of off-diagonal pivots): 19
>>         INFOG(13) (number of delayed pivots after factorization): 8
>>         INFOG(14) (number of memory compress after factorization): 0
>>         INFOG(15) (number of steps of iterative refinement after
>> solution): 0
>>         INFOG(16) (estimated size (in MB) of all MUMPS internal data for
>> factorization after analysis: value on the most memory consuming
>> processor): 7
>>         INFOG(17) (estimated size of all MUMPS internal data for
>> factorization after analysis: sum over all processors): 7
>>         INFOG(18) (size of all MUMPS internal data allocated during
>> factorization: value on the most memory consuming processor): 7
>>         INFOG(19) (size of all MUMPS internal data allocated during
>> factorization: sum over all processors): 7
>>         INFOG(20) (estimated number of entries in the factors): 255801
>>         INFOG(21) (size in MB of memory effectively used during
>> factorization - value on the most memory consuming processor): 7
>>         INFOG(22) (size in MB of memory effectively used during
>> factorization - sum over all processors): 7
>>         INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0
>>
>>         INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1
>>
>>         INFOG(25) (after factorization: number of pivots modified by
>> static pivoting): 0
>>         INFOG(28) (after factorization: number of null pivots
>> encountered): 0
>>         INFOG(29) (after factorization: effective number of entries in
>> the factors (sum over all processors)): 255865
>>         INFOG(30, 31) (after solution: size in Mbytes of memory used
>> during solution phase): 5, 5
>>         INFOG(32) (after analysis: type of analysis done): 1
>>         INFOG(33) (value used for ICNTL(8)): 7
>>         INFOG(34) (exponent of the determinant if determinant is
>> requested): 0
>> linear system matrix = precond matrix:
>> Mat Object:     1 MPI processes
>> type: seqaij
>> rows=15991, cols=15991
>> total: nonzeros=223820, allocated nonzeros=431698
>> total number of mallocs used during MatSetValues calls =15991
>> using I-node routines: found 4000 nodes, limit used is 5
>>
>>
>>
>>
>> -gideon
>>
>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>> Barry,
>>
>> I finally got a chance to really try using the grid sequencing within my
>> code.  I find that, in some cases, even if it can solve successfully on the
>> coarsest mesh, the SNES fails, usually due to a line search failure, when
>> it tries to compute along the grid sequence.  Would you have any
>> suggestions?
>>
>> I apologize if I have asked before, but can you give me -snes_view for
>> the solver? I could not find it in the email thread.
>>
>> I would suggest trying to fiddle with the line search, or precondition it
>> with Richardson. It would be nice to see -snes_monitor
>> for the runs that fail, and then we can break down the residual into
>> fields and look at it again (if my custom residual monitor
>> does not work we can write one easily). Seeing which part of the residual
>> does not converge is key to designing the NASM
>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai,
>> present it. We need better monitoring in PETSc.
>>
>> Thanks,
>>
>> Matt
>>
>> -gideon
>>
>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> Yes, if i continue in this parameter on the coarse mesh, I can generally
>> solve at all values. I do find that I need to do some amount of
>> continuation to solve near the endpoint.  The problem is that on the coarse
>> mesh, things are not fully resolved at all the values along the
>> continuation parameter, and I would like to do refinement.
>>
>> One subtlety is that I actually want the intermediate continuation
>> solutions  too.  Currently, without doing any grid sequence, I compute
>> each, write it to disk, and then go on to the next one.  So I now need to
>> go back an refine them.  I was thinking that perhaps I could refine them on
>> the fly, dump them to disk, and use the coarse solution as the starting
>> guess at the next iteration, but that would seem to require resetting the
>> snes back to the coarse grid.
>>
>> The alternative would be to just script the mesh refinement in a post
>> processing stage, where each value of the continuation is parameter is
>> loaded on the coarse mesh, and refined.  Perhaps that?s the most practical
>> thing to do.
>>
>>
>> I would do the following. Create your DM and create a SNES that will do
>> the continuation
>>
>> loop over continuation parameter
>>
>> SNESSolve(snes,NULL,Ucoarse);
>>
>> if (you decide you want to see the refined solution at this continuation
>> point) {
>>     SNESCreate(comm,&snesrefine);
>>     SNESSetDM()
>>     etc
>>     SNESSetGridSequence(snesrefine,)
>>     SNESSolve(snesrefine,0,Ucoarse);
>>     SNESGetSolution(snesrefine,&Ufine);
>>     VecView(Ufine or do whatever you want to do with the Ufine at that
>> continuation point
>>     SNESDestroy(snesrefine);
>> end if
>>
>> end loop over continuation parameter.
>>
>> Barry
>>
>>
>> -gideon
>>
>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>>
>> 3.  This problem is actually part of a continuation problem that roughly
>> looks like this
>>
>> for( continuation parameter p = 0 to 1){
>>
>> solve with parameter p_i using solution from p_{i-1},
>> }
>>
>> What I would like to do is to start the solver, for each value of
>> parameter p_i on the coarse mesh, and then do grid sequencing on that.  But
>> it appears that after doing grid sequencing on the initial p_0 = 0, the
>> SNES is set to use the finer mesh.
>>
>>
>> So you are using continuation to give you a good enough initial guess on
>> the coarse level to even get convergence on the coarse level? First I would
>> check if you even need the continuation (or can you not even solve the
>> coarse problem without it).
>>
>> If you do need the continuation then you will need to tweak how you do
>> the grid sequencing. I think this will work:
>>
>> Do not use -snes_grid_sequencing
>>
>> Run SNESSolve() as many times as you want with your continuation
>> parameter. This will all happen on the coarse mesh.
>>
>> Call SNESSetGridSequence()
>>
>> Then call SNESSolve() again and it will do one solve on the coarse level
>> and then interpolate to the next level etc.
>>
>>
>>
>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/43045819/attachment-0001.html>

From knepley at gmail.com  Tue Sep  8 20:58:45 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Sep 2015 20:58:45 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
Message-ID: <CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>

On Tue, Sep 8, 2015 at 8:51 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Going back to this point, a case that previously worked, is now giving me
> problems that I don?t understand:
>
>   0 SNES Function norm 1.421454390131e-02
>     0 KSP preconditioned resid norm 7.635339584553e-03 true resid norm
> 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.838956718317e-07 true resid norm
> 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
>     2 KSP preconditioned resid norm 5.889748199467e-12 true resid norm
> 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
>   1 SNES Function norm 4.476139939604e-06
>     0 KSP preconditioned resid norm 4.897102387844e-05 true resid norm
> 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.133649519960e-09 true resid norm
> 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
>     2 KSP preconditioned resid norm 2.223328042462e-14 true resid norm
> 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
>   2 SNES Function norm 5.447761863330e-09
>     0 KSP preconditioned resid norm 9.703890346186e-09 true resid norm
> 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.982949185218e-13 true resid norm
> 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
>     2 KSP preconditioned resid norm 7.045835741315e-18 true resid norm
> 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
> Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT iterations
> 2
>

Somehow the maximum number of function evaluations is being exceeded. Oh,
are you using -snes_fd? This uses a function evaluation
for each column, so you might have to up it > 10000. The convergence is
fine.

  Thanks,

     Matt


> -gideon
>
> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>   You can run with -snes_fd it will be slow so run with only two levels of
> grid sequencing; check the convergence rate of Newton on the finer grid. It
> will hopefully be much better (also KSP should converge in one iteration).
> Send the output
>
>  Barry
>
> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> I went through my code and nothing jumps out at me as clearly wrong.  Is
> there a way to dump to disk vectors and matrices that would be useful for
> the comparison?  I imagine I?d want the jacobian that I am specifying, but
> what would I want to compare that against in terms of using
> -snes_mf_operator?
>
>
> -gideon
>
> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> Just to clarify, again, during the linear solve phase, even though the PC
> residual norm was small (getting to 10^{-10} in 2-3 iterations) that the
> true residual norm could be comparatively large signified there is
> something wrong.  The one thing i was curious about is why it seems that
> there is only an issue during grid sequencing.  I?m not seeing that kind of
> behavior on the coarsest mesh.
>
> That definitely sounds like a bug in the Jacobian evaluation. Maybe you
> assume a certain mesh size, and when
> PETSc refines the DA and passes in the new one something goes wrong?
>
>   Matt
>
> -gideon
>
> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Alright, I?ll take another look at what has been implemented in my
> Jacobian file.  One thing I?m a bit unclear about is do I need these
>  -snes_mf and snes_mf_operator flags if I have, coded in,
>
> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>
>
> Gideon,
>
>  In your case the DM creates that Jacobian matrix object and
> form_function_jacobian's job is just to fill it up with the correct values
> for the current solution value.   If you use -snes_mf_operator then SNES
> uses a matrix-free multiple for the first matrix and your Jacobian to build
> the preconditioner. With -snes_mf it uses a matrix-free multiple for the
> first and second matrix and ignores what you compute. Since it is matrix
> free in both it cannot build a preconditioner hence the -pc_type none
>
> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian
> you provide is correct
>
>
> Barry
>
>
>
> -gideon
>
> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> If the Jacobian you provide is correct then using -snes_mf with a PC type
> of LU should give two iterations (or at most three) to get very good
> (10^-12) convergence of the linear system. Here you are not getting that
> after a few SNES iterations. So I am strongly leaning to something wrong
> with the Jacobian you provide.  -snes_type test unfortunately doesn't catch
> all problems.
>
> You can try -snes_mf -pc_type none and see how the linear solver
> converges.
>
>
> Barry
>
>
>
> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Off the top of my head, I can?t rule out a null space in this problem. I
> thought I did a pretty good job of coding the Jacobian by hand.  And before
> I made that change to the Jacobian file, changing J for Jpre, if I ran with
>  -snes_check_jacobian, it appeared to be satisfactory.
>
> When I run with -ksp_type fgmres, I get
>
>   0 SNES Function norm 3.857327250267e-09
>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm
> 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm
> 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm
> 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>   1 SNES Function norm 2.479697868412e-12
> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
> 0 SNES Function norm 5.066222213176e+03
>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm
> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm
> 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
> 1 SNES Function norm 8.484696862318e+02
>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm
> 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm
> 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm
> 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
> 2 SNES Function norm 6.551139809777e+02
>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm
> 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm
> 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm
> 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
> 3 SNES Function norm 5.771483337231e+02
>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm
> 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm
> 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm
> 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
> 4 SNES Function norm 5.237540245670e+02
>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm
> 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm
> 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm
> 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
> 5 SNES Function norm 4.753058321803e+02
>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm
> 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm
> 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm
> 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
> 6 SNES Function norm 4.221482124411e+02
>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm
> 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm
> 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm
> 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
> 7 SNES Function norm 3.803620028964e+02
>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm
> 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm
> 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm
> 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
> 8 SNES Function norm 3.769519577570e+02
>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm
> 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm
> 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm
> 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm
> 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
> 9 SNES Function norm 3.766389203406e+02
>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm
> 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm
> 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm
> 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
> 10 SNES Function norm 3.751822489648e+02
>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm
> 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm
> 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm
> 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
> 11 SNES Function norm 3.737189963925e+02
>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm
> 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm
> 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm
> 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm
> 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
> 12 SNES Function norm 3.737110331594e+02
>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm
> 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm
> 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm
> 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
> 13 SNES Function norm 3.684175513154e+02
>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm
> 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm
> 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm
> 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
> 14 SNES Function norm 3.509930783933e+02
>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm
> 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm
> 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm
> 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
> 15 SNES Function norm 3.162121096565e+02
>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm
> 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm
> 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm
> 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
> 16 SNES Function norm 2.890165052818e+02
>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm
> 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm
> 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm
> 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
> 17 SNES Function norm 2.816929421445e+02
>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm
> 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm
> 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm
> 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm
> 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
> 18 SNES Function norm 2.814407559738e+02
>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm
> 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm
> 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm
> 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
> 19 SNES Function norm 2.810261821835e+02
>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm
> 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm
> 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm
> 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
> 20 SNES Function norm 2.755723672077e+02
>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm
> 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm
> 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm
> 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm
> 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm
> 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm
> 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm
> 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
> 21 SNES Function norm 2.755720389981e+02
>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm
> 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm
> 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm
> 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm
> 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm
> 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm
> 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm
> 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
> 22 SNES Function norm 2.755716464881e+02
>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm
> 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm
> 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm
> 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm
> 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm
> 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
> 23 SNES Function norm 2.755713163168e+02
>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm
> 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm
> 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm
> 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm
> 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm
> 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm
> 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm
> 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm
> 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm
> 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
> 24 SNES Function norm 2.755712363532e+02
>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm
> 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm
> 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm
> 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm
> 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm
> 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm
> 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm
> 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm
> 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm
> 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm
> 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm
> 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm
> 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm
> 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm
> 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm
> 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm
> 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm
> 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm
> 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm
> 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm
> 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm
> 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm
> 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm
> 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm
> 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm
> 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm
> 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm
> 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm
> 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm
> 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm
> 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm
> 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm
> 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm
> 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm
> 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm
> 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm
> 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm
> 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm
> 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm
> 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm
> 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm
> 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm
> 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm
> 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm
> 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm
> 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm
> 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm
> 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm
> 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm
> 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm
> 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm
> 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm
> 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm
> 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm
> 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm
> 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm
> 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm
> 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm
> 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm
> 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm
> 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm
> 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm
> 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm
> 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm
> 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm
> 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm
> 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm
> 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm
> 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm
> 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm
> 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm
> 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm
> 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm
> 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm
> 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm
> 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm
> 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm
> 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm
> 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm
> 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm
> 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm
> 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm
> 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm
> 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm
> 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm
> 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm
> 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm
> 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm
> 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm
> 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm
> 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm
> 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm
> 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm
> 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm
> 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm
> 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm
> 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm
> 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm
> 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm
> 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm
> 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm
> 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm
> 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm
> 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm
> 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm
> 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm
> 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm
> 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm
> 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm
> 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm
> 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm
> 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm
> 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm
> 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm
> 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm
> 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm
> 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm
> 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm
> 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm
> 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm
> 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm
> 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm
> 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm
> 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm
> 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm
> 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm
> 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm
> 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm
> 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm
> 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm
> 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm
> 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm
> 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm
> 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm
> 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm
> 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm
> 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm
> 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm
> 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm
> 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm
> 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm
> 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm
> 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm
> 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm
> 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm
> 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm
> 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm
> 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm
> 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm
> 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm
> 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm
> 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm
> 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm
> 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm
> 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm
> 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm
> 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm
> 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm
> 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm
> 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm
> 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm
> 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm
> 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm
> 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm
> 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm
> 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm
> 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm
> 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm
> 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm
> 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm
> 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm
> 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm
> 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm
> 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm
> 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm
> 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm
> 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm
> 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm
> 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm
> 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm
> 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm
> 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm
> 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm
> 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm
> 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm
> 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm
> 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm
> 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm
> 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm
> 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm
> 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm
> 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm
> 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm
> 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm
> 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm
> 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm
> 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm
> 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm
> 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm
> 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm
> 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm
> 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm
> 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm
> 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm
> 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm
> 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm
> 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm
> 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm
> 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm
> 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm
> 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm
> 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>
> -gideon
>
> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Is there any chance your Jacobian may have a null space? (For example with
> Neumann boundary conditions there is often a nullspace)
>
> If not run with with -ksp_type fgmres and send the same output.
>
> Something is screwy with the linear system convergence, could be several
> things including either a null space problem or that your Jacobian that you
> provide is terribly wrong.
>
> Barry
>
>
> 21 SNES Function norm 2.987064584431e+02
>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm
> 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm
> 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm
> 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>
>
> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Got it, fixing that, and returning to the original question, this is what
> I now get, when I use those two flags:
>
>  0 SNES Function norm 1.132185384796e-08
>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm
> 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm
> 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>  1 SNES Function norm 2.177599365111e-12
> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
> 0 SNES Function norm 5.066222213176e+03
>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm
> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm
> 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
> 1 SNES Function norm 8.482593852817e+02
>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm
> 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm
> 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm
> 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
> 2 SNES Function norm 6.543140468549e+02
>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm
> 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm
> 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm
> 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
> 3 SNES Function norm 5.766430557220e+02
>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm
> 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm
> 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm
> 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
> 4 SNES Function norm 5.235211958260e+02
>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm
> 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm
> 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm
> 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
> 5 SNES Function norm 4.752913229649e+02
>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm
> 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm
> 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm
> 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
> 6 SNES Function norm 4.220255380391e+02
>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm
> 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm
> 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm
> 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
> 7 SNES Function norm 3.805408907074e+02
>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm
> 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm
> 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
> 8 SNES Function norm 3.764619752339e+02
>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm
> 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm
> 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm
> 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
> 9 SNES Function norm 3.761182227091e+02
>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm
> 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm
> 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm
> 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
> 10 SNES Function norm 3.740017190063e+02
>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm
> 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm
> 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm
> 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
> 11 SNES Function norm 3.725903477238e+02
>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm
> 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm
> 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm
> 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
> 12 SNES Function norm 3.716162097231e+02
>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm
> 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm
> 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm
> 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
> 13 SNES Function norm 3.674168632847e+02
>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm
> 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm
> 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm
> 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
> 14 SNES Function norm 3.532395445266e+02
>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm
> 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm
> 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm
> 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
> 15 SNES Function norm 3.182438872366e+02
>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm
> 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm
> 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm
> 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
> 16 SNES Function norm 3.091759892779e+02
>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm
> 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm
> 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm
> 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
> 17 SNES Function norm 2.987839504359e+02
>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm
> 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm
> 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm
> 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
> 18 SNES Function norm 2.987073622777e+02
>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm
> 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm
> 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm
> 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
> 19 SNES Function norm 2.987067936734e+02
>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm
> 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm
> 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm
> 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
> 20 SNES Function norm 2.987067502910e+02
>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm
> 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm
> 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm
> 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
> 21 SNES Function norm 2.987064584431e+02
>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm
> 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm
> 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm
> 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
> 22 SNES Function norm 2.987064525262e+02
>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm
> 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm
> 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm
> 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
> 23 SNES Function norm 2.987064121622e+02
>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm
> 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm
> 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm
> 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
> 24 SNES Function norm 2.987063973426e+02
>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm
> 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm
> 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm
> 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
> 25 SNES Function norm 2.987063920553e+02
>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm
> 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm
> 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm
> 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
> 26 SNES Function norm 2.987063919786e+02
>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm
> 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm
> 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm
> 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
> 27 SNES Function norm 2.987063393145e+02
>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm
> 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm
> 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm
> 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
> 28 SNES Function norm 2.987063253277e+02
>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm
> 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm
> 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm
> 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
> 29 SNES Function norm 2.987063197153e+02
>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm
> 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm
> 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm
> 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
> 30 SNES Function norm 2.987063193089e+02
>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm
> 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm
> 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm
> 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
> 31 SNES Function norm 2.987063192061e+02
>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm
> 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm
> 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm
> 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
> 32 SNES Function norm 2.987063094492e+02
>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm
> 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm
> 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm
> 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
> 33 SNES Function norm 2.987063089942e+02
>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm
> 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm
> 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm
> 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
> 34 SNES Function norm 2.987062660109e+02
>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm
> 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm
> 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm
> 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
> 35 SNES Function norm 2.987062107900e+02
>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm
> 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm
> 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm
> 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
> 36 SNES Function norm 2.987062055224e+02
>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm
> 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm
> 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm
> 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
> 37 SNES Function norm 2.987061774798e+02
>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm
> 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm
> 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm
> 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
> 38 SNES Function norm 2.987061715400e+02
>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm
> 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm
> 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm
> 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
> 39 SNES Function norm 2.987061699634e+02
>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm
> 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm
> 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm
> 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
> 40 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm
> 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
> 41 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm
> 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
> 42 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm
> 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
> 43 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm
> 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
> 44 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm
> 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
> 45 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm
> 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
> 46 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm
> 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
> 47 SNES Function norm 2.987061630064e+02
>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm
> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm
> 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>
>
> -gideon
>
> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> This indicates that somewhere in your ComputeJacobian you are setting
> matrix entries with the first Mat argument when you should always set them
> with the second matrix argument. For example if you have
>
> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>
> you should call all the MatSetValues() with jpre, no J. Then at the end of
> the function you should call MatAssemblyBegin/End() on jpre then on J if J
> is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>
> This is a minor glitch we'll get past.
>
> Barry
>
> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> By the way, I tried using a different petsc installation, and now, rather
> than the segmentation fault, I get the following error:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Mat type mffd
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named
> gs_air by gideon Mon Sep  7 21:32:18 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes
> --download-suitesparse=yes --download-superlu=yes
> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes
> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes
> --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in
> /opt/petsc-3.5.4/src/mat/interface/matrix.c
>
> -gideon
>
> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Hmm,
>
> Ok you can try running it directly in the debugger since it is one
> process, type
>
> gdb ./blowup_batch_refine
>
> then
>
> when the debugger comes up (if it does not cut and paste all output and
> send it)
>
> run -on_error_abort -snes_mf_operator  and any other options you normally
> use
>
>
> Barry
>
> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Running with that flag gives me this:
>
> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111
> on gs_air
> Unable to start debugger: No such file or directory
>
>
>
> -gideon
>
> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> This should not happen. Run with a debug version of PETSc installed and
> the option -start_in_debugger noxterm  Once the debugger starts up type
> cont and when it crashes type where or bt  Send all output
>
>
>
> Barry
>
>
> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> I?m getting an error with -snes_mf_operator,
>
> 0 SNES Function norm 1.421454390131e-02
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by
> gideon Mon Sep  7 21:08:19 2015
> [0]PETSC ERROR: Configure options --prefix=/opt/local
> --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries
> --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0
> --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate
> --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local
> --with-superlu-dir=/opt/local --with-metis-dir=/opt/local
> --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local
> --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local
> CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp
> FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp
> F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os
> FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names"
> CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os
> FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports
> --with-mpiexec=mpiexec-mpich-mp
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>
> -gideon
>
> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc
> SNES is generating a poor descent direction. You can try
> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What
> happens?
>
> Barry
>
>
>
> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> No problem Matt, I don?t think we had previously discussed that output.
> Here is a case where things fail.
>
> 0 SNES Function norm 4.027481756921e-09
> 1 SNES Function norm 1.760477878365e-12
> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
> 0 SNES Function norm 5.066222213176e+03
> 1 SNES Function norm 8.484697184230e+02
> 2 SNES Function norm 6.549559723294e+02
> 3 SNES Function norm 5.770723278153e+02
> 4 SNES Function norm 5.237702240594e+02
> 5 SNES Function norm 4.753909019848e+02
> 6 SNES Function norm 4.221784590755e+02
> 7 SNES Function norm 3.806525080483e+02
> 8 SNES Function norm 3.762054656019e+02
> 9 SNES Function norm 3.758975226873e+02
> 10 SNES Function norm 3.757032042706e+02
> 11 SNES Function norm 3.728798164234e+02
> 12 SNES Function norm 3.723078741075e+02
> 13 SNES Function norm 3.721848059825e+02
> 14 SNES Function norm 3.720227575629e+02
> 15 SNES Function norm 3.720051998555e+02
> 16 SNES Function norm 3.718945430587e+02
> 17 SNES Function norm 3.700412694044e+02
> 18 SNES Function norm 3.351964889461e+02
> 19 SNES Function norm 3.096016086233e+02
> 20 SNES Function norm 3.008410789787e+02
> 21 SNES Function norm 2.752316716557e+02
> 22 SNES Function norm 2.707658474165e+02
> 23 SNES Function norm 2.698436736049e+02
> 24 SNES Function norm 2.618233857172e+02
> 25 SNES Function norm 2.600121920634e+02
> 26 SNES Function norm 2.585046423168e+02
> 27 SNES Function norm 2.568551090220e+02
> 28 SNES Function norm 2.556404537064e+02
> 29 SNES Function norm 2.536353523683e+02
> 30 SNES Function norm 2.533596070171e+02
> 31 SNES Function norm 2.532324379596e+02
> 32 SNES Function norm 2.531842335211e+02
> 33 SNES Function norm 2.531684527520e+02
> 34 SNES Function norm 2.531637604618e+02
> 35 SNES Function norm 2.531624767821e+02
> 36 SNES Function norm 2.531621359093e+02
> 37 SNES Function norm 2.531620504925e+02
> 38 SNES Function norm 2.531620350055e+02
> 39 SNES Function norm 2.531620310522e+02
> 40 SNES Function norm 2.531620300471e+02
> 41 SNES Function norm 2.531620298084e+02
> 42 SNES Function norm 2.531620297478e+02
> 43 SNES Function norm 2.531620297324e+02
> 44 SNES Function norm 2.531620297303e+02
> 45 SNES Function norm 2.531620297302e+02
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
> 0 SNES Function norm 9.636339304380e+03
> 1 SNES Function norm 8.997731184634e+03
> 2 SNES Function norm 8.120498349232e+03
> 3 SNES Function norm 7.322379894820e+03
> 4 SNES Function norm 6.599581599149e+03
> 5 SNES Function norm 6.374872854688e+03
> 6 SNES Function norm 6.372518007653e+03
> 7 SNES Function norm 6.073996314301e+03
> 8 SNES Function norm 5.635965277054e+03
> 9 SNES Function norm 5.155389064046e+03
> 10 SNES Function norm 5.080567902638e+03
> 11 SNES Function norm 5.058878643969e+03
> 12 SNES Function norm 5.058835649793e+03
> 13 SNES Function norm 5.058491285707e+03
> 14 SNES Function norm 5.057452865337e+03
> 15 SNES Function norm 5.057226140688e+03
> 16 SNES Function norm 5.056651272898e+03
> 17 SNES Function norm 5.056575190057e+03
> 18 SNES Function norm 5.056574632598e+03
> 19 SNES Function norm 5.056574520229e+03
> 20 SNES Function norm 5.056574492569e+03
> 21 SNES Function norm 5.056574485124e+03
> 22 SNES Function norm 5.056574483029e+03
> 23 SNES Function norm 5.056574482427e+03
> 24 SNES Function norm 5.056574482302e+03
> 25 SNES Function norm 5.056574482287e+03
> 26 SNES Function norm 5.056574482282e+03
> 27 SNES Function norm 5.056574482281e+03
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
> SNES Object: 1 MPI processes
> type: newtonls
> maximum iterations=50, maximum function evaluations=10000
> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
> total number of linear solver iterations=28
> total number of function evaluations=323
> total number of grid sequence refinements=2
> SNESLineSearch Object:   1 MPI processes
> type: bt
> interpolation: cubic
> alpha=1.000000e-04
> maxstep=1.000000e+08, minlambda=1.000000e-12
> tolerances: relative=1.000000e-08, absolute=1.000000e-15,
> lambda=1.000000e-08
> maximum iterations=40
> KSP Object:   1 MPI processes
> type: gmres
> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=10000, initial guess is zero
> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using PRECONDITIONED norm type for convergence test
> PC Object:   1 MPI processes
> type: lu
> LU: out-of-place factorization
> tolerance for zero pivot 2.22045e-14
> matrix ordering: nd
> factor fill ratio given 0, needed 0
> Factored matrix follows:
>  Mat Object:           1 MPI processes
>    type: seqaij
>    rows=15991, cols=15991
>    package used to perform factorization: mumps
>    total: nonzeros=255801, allocated nonzeros=255801
>    total number of mallocs used during MatSetValues calls =0
>      MUMPS run parameters:
>        SYM (matrix type):                   0
>        PAR (host participation):            1
>        ICNTL(1) (output for error):         6
>        ICNTL(2) (output of diagnostic msg): 0
>        ICNTL(3) (output for global info):   0
>        ICNTL(4) (level of printing):        0
>        ICNTL(5) (input mat struct):         0
>        ICNTL(6) (matrix prescaling):        7
>        ICNTL(7) (sequentia matrix ordering):6
>        ICNTL(8) (scalling strategy):        77
>        ICNTL(10) (max num of refinements):  0
>        ICNTL(11) (error analysis):          0
>        ICNTL(12) (efficiency control):                         1
>        ICNTL(13) (efficiency control):                         0
>        ICNTL(14) (percentage of estimated workspace increase): 20
>        ICNTL(18) (input mat struct):                           0
>        ICNTL(19) (Shur complement info):                       0
>        ICNTL(20) (rhs sparse pattern):                         0
>        ICNTL(21) (somumpstion struct):                            0
>        ICNTL(22) (in-core/out-of-core facility):               0
>        ICNTL(23) (max size of memory can be allocated locally):0
>        ICNTL(24) (detection of null pivot rows):               0
>        ICNTL(25) (computation of a null space basis):          0
>        ICNTL(26) (Schur options for rhs or solution):          0
>        ICNTL(27) (experimental parameter):                     -8
>        ICNTL(28) (use parallel or sequential ordering):        1
>        ICNTL(29) (parallel ordering):                          0
>        ICNTL(30) (user-specified set of entries in inv(A)):    0
>        ICNTL(31) (factors is discarded in the solve phase):    0
>        ICNTL(33) (compute determinant):                        0
>        CNTL(1) (relative pivoting threshold):      0.01
>        CNTL(2) (stopping criterion of refinement): 1.49012e-08
>        CNTL(3) (absomumpste pivoting threshold):      0
>        CNTL(4) (vamumpse of static pivoting):         -1
>        CNTL(5) (fixation for null pivots):         0
>        RINFO(1) (local estimated flops for the elimination after
> analysis):
>          [0] 1.95838e+06
>        RINFO(2) (local estimated flops for the assembly after
> factorization):
>          [0]  143924
>        RINFO(3) (local estimated flops for the elimination after
> factorization):
>          [0]  1.95943e+06
>        INFO(15) (estimated size of (in MB) MUMPS internal data for running
> numerical factorization):
>        [0] 7
>        INFO(16) (size of (in MB) MUMPS internal data used during numerical
> factorization):
>          [0] 7
>        INFO(23) (num of pivots eliminated on this processor after
> factorization):
>          [0] 15991
>        RINFOG(1) (global estimated flops for the elimination after
> analysis): 1.95838e+06
>        RINFOG(2) (global estimated flops for the assembly after
> factorization): 143924
>        RINFOG(3) (global estimated flops for the elimination after
> factorization): 1.95943e+06
>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>        INFOG(3) (estimated real workspace for factors on all processors
> after analysis): 255801
>        INFOG(4) (estimated integer workspace for factors on all processors
> after analysis): 127874
>        INFOG(5) (estimated maximum front size in the complete tree): 11
>        INFOG(6) (number of nodes in the complete tree): 3996
>        INFOG(7) (ordering option effectively use after analysis): 6
>        INFOG(8) (structural symmetry in percent of the permuted matrix
> after analysis): 86
>        INFOG(9) (total real/complex workspace to store the matrix factors
> after factorization): 255865
>        INFOG(10) (total integer space store the matrix factors after
> factorization): 127890
>        INFOG(11) (order of largest frontal matrix after factorization): 11
>        INFOG(12) (number of off-diagonal pivots): 19
>        INFOG(13) (number of delayed pivots after factorization): 8
>        INFOG(14) (number of memory compress after factorization): 0
>        INFOG(15) (number of steps of iterative refinement after solution):
> 0
>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for
> factorization after analysis: value on the most memory consuming
> processor): 7
>        INFOG(17) (estimated size of all MUMPS internal data for
> factorization after analysis: sum over all processors): 7
>        INFOG(18) (size of all MUMPS internal data allocated during
> factorization: value on the most memory consuming processor): 7
>        INFOG(19) (size of all MUMPS internal data allocated during
> factorization: sum over all processors): 7
>        INFOG(20) (estimated number of entries in the factors): 255801
>        INFOG(21) (size in MB of memory effectively used during
> factorization - value on the most memory consuming processor): 7
>        INFOG(22) (size in MB of memory effectively used during
> factorization - sum over all processors): 7
>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0
>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1
>        INFOG(25) (after factorization: number of pivots modified by static
> pivoting): 0
>        INFOG(28) (after factorization: number of null pivots encountered):
> 0
>        INFOG(29) (after factorization: effective number of entries in the
> factors (sum over all processors)): 255865
>        INFOG(30, 31) (after solution: size in Mbytes of memory used during
> solution phase): 5, 5
>        INFOG(32) (after analysis: type of analysis done): 1
>        INFOG(33) (value used for ICNTL(8)): 7
>        INFOG(34) (exponent of the determinant if determinant is
> requested): 0
> linear system matrix = precond matrix:
> Mat Object:     1 MPI processes
> type: seqaij
> rows=15991, cols=15991
> total: nonzeros=223820, allocated nonzeros=431698
> total number of mallocs used during MatSetValues calls =15991
> using I-node routines: found 4000 nodes, limit used is 5
>
>
>
>
> -gideon
>
> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> Barry,
>
> I finally got a chance to really try using the grid sequencing within my
> code.  I find that, in some cases, even if it can solve successfully on the
> coarsest mesh, the SNES fails, usually due to a line search failure, when
> it tries to compute along the grid sequence.  Would you have any
> suggestions?
>
> I apologize if I have asked before, but can you give me -snes_view for the
> solver? I could not find it in the email thread.
>
> I would suggest trying to fiddle with the line search, or precondition it
> with Richardson. It would be nice to see -snes_monitor
> for the runs that fail, and then we can break down the residual into
> fields and look at it again (if my custom residual monitor
> does not work we can write one easily). Seeing which part of the residual
> does not converge is key to designing the NASM
> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai,
> present it. We need better monitoring in PETSc.
>
> Thanks,
>
> Matt
>
> -gideon
>
> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Yes, if i continue in this parameter on the coarse mesh, I can generally
> solve at all values. I do find that I need to do some amount of
> continuation to solve near the endpoint.  The problem is that on the coarse
> mesh, things are not fully resolved at all the values along the
> continuation parameter, and I would like to do refinement.
>
> One subtlety is that I actually want the intermediate continuation
> solutions  too.  Currently, without doing any grid sequence, I compute
> each, write it to disk, and then go on to the next one.  So I now need to
> go back an refine them.  I was thinking that perhaps I could refine them on
> the fly, dump them to disk, and use the coarse solution as the starting
> guess at the next iteration, but that would seem to require resetting the
> snes back to the coarse grid.
>
> The alternative would be to just script the mesh refinement in a post
> processing stage, where each value of the continuation is parameter is
> loaded on the coarse mesh, and refined.  Perhaps that?s the most practical
> thing to do.
>
>
> I would do the following. Create your DM and create a SNES that will do
> the continuation
>
> loop over continuation parameter
>
> SNESSolve(snes,NULL,Ucoarse);
>
> if (you decide you want to see the refined solution at this continuation
> point) {
>    SNESCreate(comm,&snesrefine);
>    SNESSetDM()
>    etc
>    SNESSetGridSequence(snesrefine,)
>    SNESSolve(snesrefine,0,Ucoarse);
>    SNESGetSolution(snesrefine,&Ufine);
>    VecView(Ufine or do whatever you want to do with the Ufine at that
> continuation point
>    SNESDestroy(snesrefine);
> end if
>
> end loop over continuation parameter.
>
> Barry
>
>
> -gideon
>
> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>
> 3.  This problem is actually part of a continuation problem that roughly
> looks like this
>
> for( continuation parameter p = 0 to 1){
>
> solve with parameter p_i using solution from p_{i-1},
> }
>
> What I would like to do is to start the solver, for each value of
> parameter p_i on the coarse mesh, and then do grid sequencing on that.  But
> it appears that after doing grid sequencing on the initial p_0 = 0, the
> SNES is set to use the finer mesh.
>
>
> So you are using continuation to give you a good enough initial guess on
> the coarse level to even get convergence on the coarse level? First I would
> check if you even need the continuation (or can you not even solve the
> coarse problem without it).
>
> If you do need the continuation then you will need to tweak how you do the
> grid sequencing. I think this will work:
>
> Do not use -snes_grid_sequencing
>
> Run SNESSolve() as many times as you want with your continuation
> parameter. This will all happen on the coarse mesh.
>
> Call SNESSetGridSequence()
>
> Then call SNESSolve() again and it will do one solve on the coarse level
> and then interpolate to the next level etc.
>
>
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/b5c37d05/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Sep  8 21:11:50 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 Sep 2015 21:11:50 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
Message-ID: <9C85B843-DBAB-4C3D-B68E-EFC9CE5130B8@mcs.anl.gov>


  Run with something like -snes_max_funcs 1000000000 Matt is right, the -snes_fd requires many function evaluations.

  Barry


  
> On Sep 8, 2015, at 8:58 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Sep 8, 2015 at 8:51 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> Going back to this point, a case that previously worked, is now giving me problems that I don?t understand:
> 
>   0 SNES Function norm 1.421454390131e-02 
>     0 KSP preconditioned resid norm 7.635339584553e-03 true resid norm 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.838956718317e-07 true resid norm 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
>     2 KSP preconditioned resid norm 5.889748199467e-12 true resid norm 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
>   1 SNES Function norm 4.476139939604e-06 
>     0 KSP preconditioned resid norm 4.897102387844e-05 true resid norm 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.133649519960e-09 true resid norm 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
>     2 KSP preconditioned resid norm 2.223328042462e-14 true resid norm 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
>   2 SNES Function norm 5.447761863330e-09 
>     0 KSP preconditioned resid norm 9.703890346186e-09 true resid norm 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.982949185218e-13 true resid norm 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
>     2 KSP preconditioned resid norm 7.045835741315e-18 true resid norm 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
> Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT iterations 2
> 
> Somehow the maximum number of function evaluations is being exceeded. Oh, are you using -snes_fd? This uses a function evaluation
> for each column, so you might have to up it > 10000. The convergence is fine.
> 
>   Thanks,
> 
>      Matt
>  
> -gideon
> 
>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>> 
>>  Barry
>> 
>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>> 
>>> 
>>> -gideon
>>> 
>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>> 
>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>> 
>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>> 
>>>>   Matt
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>> 
>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>> 
>>>>> Gideon,
>>>>> 
>>>>>  In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>> 
>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>> 
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>> 
>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>> 
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>> 
>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>> 
>>>>>>>>   0 SNES Function norm 3.857327250267e-09 
>>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>   1 SNES Function norm 2.479697868412e-12 
>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>> 
>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>> 
>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>> 
>>>>>>>>>>  0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>  1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>> 
>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>> 
>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>> 
>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>> 
>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>> 
>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>> 
>>>>>>>>>>>>> then 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>    type: seqaij
>>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>      MUMPS run parameters:
>>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>        PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>          [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>          [0]  143924 
>>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>          [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>          [0] 15991 
>>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>    SNESSetDM()
>>>>>>>>>>>>>>>>>>>>    etc
>>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From gideon.simpson at gmail.com  Tue Sep  8 21:48:07 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 22:48:07 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
Message-ID: <360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>

Your comment about the different pieces scaling different made me think of something, and I made a slight modification.  It still fails, but the conditioning looks much better, no?


      0 SNES Function norm 4.027474333769e-09 
        0 KSP preconditioned resid norm 9.390682304142e-09 true resid norm 4.027474333769e-09 ||r(i)||/||b|| 1.000000000000e+00
        1 KSP preconditioned resid norm 4.937459249116e-20 true resid norm 3.737772204700e-19 ||r(i)||/||b|| 9.280685350023e-11
      1 SNES Function norm 2.666655509899e-12 
    Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
    0 SNES Function norm 5.066222213175e+03 
      0 KSP preconditioned resid norm 3.135087050015e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.220576053756e-11 true resid norm 1.206309587478e-07 ||r(i)||/||b|| 2.381083056997e-11
    1 SNES Function norm 8.484697184819e+02 
      0 KSP preconditioned resid norm 2.668083368240e+02 true resid norm 8.484697184819e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.388166746665e-10 true resid norm 2.735732864000e-08 ||r(i)||/||b|| 3.224314085004e-11
    2 SNES Function norm 6.549559723821e+02 
      0 KSP preconditioned resid norm 1.552154273063e+02 true resid norm 6.549559723821e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 5.740227244709e-10 true resid norm 1.800722435354e-08 ||r(i)||/||b|| 2.749379364852e-11
    3 SNES Function norm 5.770723278635e+02 
      0 KSP preconditioned resid norm 1.348596653013e+02 true resid norm 5.770723278635e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.142365551700e-10 true resid norm 7.387664544933e-09 ||r(i)||/||b|| 1.280197331985e-11
    4 SNES Function norm 5.237702241066e+02 
      0 KSP preconditioned resid norm 1.524401223470e+02 true resid norm 5.237702241066e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.168592748079e-10 true resid norm 9.512653260875e-09 ||r(i)||/||b|| 1.816188248788e-11
    5 SNES Function norm 4.753909020302e+02 
      0 KSP preconditioned resid norm 1.864261890160e+02 true resid norm 4.753909020302e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.744912816503e-10 true resid norm 1.889384889937e-08 ||r(i)||/||b|| 3.974381675939e-11
    6 SNES Function norm 4.221784591168e+02 
      0 KSP preconditioned resid norm 2.550992140633e+02 true resid norm 4.221784591168e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.697718851690e-10 true resid norm 1.666255394677e-08 ||r(i)||/||b|| 3.946803439861e-11
    7 SNES Function norm 3.806525080909e+02 
      0 KSP preconditioned resid norm 5.183605288372e+02 true resid norm 3.806525080909e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.240268561213e-09 true resid norm 1.041310587019e-07 ||r(i)||/||b|| 2.735593657958e-10
    8 SNES Function norm 3.762054656363e+02 
      0 KSP preconditioned resid norm 5.320776249886e+03 true resid norm 3.762054656363e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.328856456273e-09 true resid norm 7.860313076537e-08 ||r(i)||/||b|| 2.089367059897e-10
    9 SNES Function norm 3.758975227207e+02 
      0 KSP preconditioned resid norm 6.252711400154e+03 true resid norm 3.758975227207e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.900963654627e-09 true resid norm 1.347299589292e-07 ||r(i)||/||b|| 3.584220453330e-10
   10 SNES Function norm 3.757032042008e+02 
      0 KSP preconditioned resid norm 5.890497230519e+02 true resid norm 3.757032042008e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.497822807230e-09 true resid norm 3.605311757144e-08 ||r(i)||/||b|| 9.596169840535e-11
   11 SNES Function norm 3.728798164183e+02 
      0 KSP preconditioned resid norm 1.081356468099e+03 true resid norm 3.728798164183e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.713971332514e-09 true resid norm 3.705072372867e-08 ||r(i)||/||b|| 9.936371478768e-11
   12 SNES Function norm 3.723078736486e+02 
      0 KSP preconditioned resid norm 3.745123875883e+03 true resid norm 3.723078736486e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.885886422519e-08 true resid norm 2.837206039485e-07 ||r(i)||/||b|| 7.620591022374e-10
   13 SNES Function norm 3.721848051981e+02 
      0 KSP preconditioned resid norm 3.495480420016e+03 true resid norm 3.721848051981e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.874759607493e-08 true resid norm 2.751735241072e-07 ||r(i)||/||b|| 7.393464759014e-10
   14 SNES Function norm 3.720227571463e+02 
      0 KSP preconditioned resid norm 1.265998097646e+04 true resid norm 3.720227571463e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 5.125038941589e-08 true resid norm 5.141611324046e-07 ||r(i)||/||b|| 1.382069033488e-09
   15 SNES Function norm 3.720051993666e+02 
      0 KSP preconditioned resid norm 2.912369751490e+03 true resid norm 3.720051993666e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.170907138699e-09 true resid norm 3.623872061163e-08 ||r(i)||/||b|| 9.741455407971e-11
   16 SNES Function norm 3.718945615434e+02 
      0 KSP preconditioned resid norm 1.082053028195e+03 true resid norm 3.718945615434e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.660982704773e-09 true resid norm 9.903133228018e-08 ||r(i)||/||b|| 2.662887348210e-10
   17 SNES Function norm 3.700412140239e+02 
      0 KSP preconditioned resid norm 3.676306553630e+01 true resid norm 3.700412140239e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.389666775474e-10 true resid norm 4.384613797840e-10 ||r(i)||/||b|| 1.184898771183e-12
   18 SNES Function norm 3.351961239413e+02 
      0 KSP preconditioned resid norm 1.071519127966e+02 true resid norm 3.351961239413e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.391049839274e-11 true resid norm 1.771280721778e-09 ||r(i)||/||b|| 5.284311468017e-12
   19 SNES Function norm 3.096040290055e+02 
      0 KSP preconditioned resid norm 3.362840348158e+02 true resid norm 3.096040290055e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.847034649159e-10 true resid norm 3.697317355249e-09 ||r(i)||/||b|| 1.194208411023e-11
   20 SNES Function norm 3.008687462984e+02 
      0 KSP preconditioned resid norm 7.161404453567e+01 true resid norm 3.008687462984e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.352839847013e-11 true resid norm 7.837384020140e-10 ||r(i)||/||b|| 2.604917963917e-12
   21 SNES Function norm 2.752608806757e+02 
      0 KSP preconditioned resid norm 1.727563200247e+02 true resid norm 2.752608806757e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 5.663026698374e-10 true resid norm 5.807801532466e-09 ||r(i)||/||b|| 2.109926233692e-11
   22 SNES Function norm 2.707559177016e+02 
      0 KSP preconditioned resid norm 4.125046548626e+02 true resid norm 2.707559177016e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.109007703208e-10 true resid norm 5.875389803064e-09 ||r(i)||/||b|| 2.169994972940e-11
   23 SNES Function norm 2.698855074973e+02 
      0 KSP preconditioned resid norm 7.414116220947e+01 true resid norm 2.698855074973e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.222534270039e-11 true resid norm 6.928234778488e-10 ||r(i)||/||b|| 2.567101450810e-12
   24 SNES Function norm 2.616892369202e+02 
      0 KSP preconditioned resid norm 1.819161466540e+02 true resid norm 2.616892369202e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.013662651542e-09 true resid norm 4.195842561304e-09 ||r(i)||/||b|| 1.603368411588e-11
   25 SNES Function norm 2.600248325147e+02 
      0 KSP preconditioned resid norm 1.265123744997e+02 true resid norm 2.600248325147e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.572744639399e-09 true resid norm 7.386966838391e-09 ||r(i)||/||b|| 2.840869761150e-11
   26 SNES Function norm 2.593065715720e+02 
      0 KSP preconditioned resid norm 1.314965938655e+02 true resid norm 2.593065715720e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.156889834113e-10 true resid norm 3.157778817609e-09 ||r(i)||/||b|| 1.217778168314e-11
   27 SNES Function norm 2.517348459439e+02 
      0 KSP preconditioned resid norm 2.618005944700e+01 true resid norm 2.517348459439e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.483924363849e-11 true resid norm 7.418968110702e-10 ||r(i)||/||b|| 2.947135936976e-12
   28 SNES Function norm 2.298635352641e+02 
      0 KSP preconditioned resid norm 8.770042104744e+01 true resid norm 2.298635352641e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.098440763657e-09 true resid norm 3.936819814733e-09 ||r(i)||/||b|| 1.712676962968e-11
   29 SNES Function norm 2.269871118080e+02 
      0 KSP preconditioned resid norm 2.285655807501e+02 true resid norm 2.269871118080e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.216116049541e-08 true resid norm 9.327420966977e-08 ||r(i)||/||b|| 4.109229327024e-10
   30 SNES Function norm 2.252105956612e+02 
      0 KSP preconditioned resid norm 8.597156834231e+01 true resid norm 2.252105956612e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.431615405953e-09 true resid norm 4.297181487549e-09 ||r(i)||/||b|| 1.908072519826e-11
   31 SNES Function norm 2.251883212710e+02 
      0 KSP preconditioned resid norm 7.066983917118e+01 true resid norm 2.251883212710e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.998573955346e-10 true resid norm 3.454505271056e-09 ||r(i)||/||b|| 1.534051700176e-11
   32 SNES Function norm 2.117895362999e+02 
      0 KSP preconditioned resid norm 3.830985324064e+02 true resid norm 2.117895362999e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.016573825110e-09 true resid norm 1.942657883557e-08 ||r(i)||/||b|| 9.172586698552e-11
   33 SNES Function norm 2.116404150058e+02 
      0 KSP preconditioned resid norm 5.329232125733e+02 true resid norm 2.116404150058e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.609537995048e-09 true resid norm 1.451505196923e-08 ||r(i)||/||b|| 6.858355465253e-11
   34 SNES Function norm 2.115566314991e+02 
      0 KSP preconditioned resid norm 6.609724236205e+02 true resid norm 2.115566314991e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.254105497370e-10 true resid norm 1.178528765891e-08 ||r(i)||/||b|| 5.570748397436e-11
   35 SNES Function norm 2.115028944726e+02 
      0 KSP preconditioned resid norm 8.217040556045e+02 true resid norm 2.115028944726e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.803324342385e-09 true resid norm 2.957366914143e-08 ||r(i)||/||b|| 1.398263093050e-10
   36 SNES Function norm 2.114773510911e+02 
      0 KSP preconditioned resid norm 1.091957913212e+03 true resid norm 2.114773510911e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.264170340401e-08 true resid norm 1.635833495717e-07 ||r(i)||/||b|| 7.735265678696e-10
   37 SNES Function norm 2.114573155596e+02 
      0 KSP preconditioned resid norm 1.295967660499e+03 true resid norm 2.114573155596e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.787403648651e-08 true resid norm 1.913077973453e-07 ||r(i)||/||b|| 9.047111793648e-10
   38 SNES Function norm 2.114434864633e+02 
      0 KSP preconditioned resid norm 1.567323979910e+03 true resid norm 2.114434864633e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.869044911030e-09 true resid norm 3.425540409553e-08 ||r(i)||/||b|| 1.620073744928e-10
   39 SNES Function norm 2.114337906450e+02 
      0 KSP preconditioned resid norm 1.835099219771e+03 true resid norm 2.114337906450e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.224665285731e-10 true resid norm 1.690151657152e-08 ||r(i)||/||b|| 7.993763210677e-11
   40 SNES Function norm 2.114267348757e+02 
      0 KSP preconditioned resid norm 2.146249211809e+03 true resid norm 2.114267348757e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.809299780773e-08 true resid norm 1.157600089098e-07 ||r(i)||/||b|| 5.475183116166e-10
   41 SNES Function norm 2.114215910347e+02 
      0 KSP preconditioned resid norm 2.510692990698e+03 true resid norm 2.114215910347e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.485943641401e-08 true resid norm 9.439839396991e-08 ||r(i)||/||b|| 4.464936315535e-10
   42 SNES Function norm 2.114206676373e+02 
      0 KSP preconditioned resid norm 3.401544811763e+03 true resid norm 2.114206676373e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.910138749741e-09 true resid norm 3.806926432018e-08 ||r(i)||/||b|| 1.800640625423e-10
   43 SNES Function norm 2.114186309697e+02 
      0 KSP preconditioned resid norm 3.955962524189e+03 true resid norm 2.114186309697e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.500201400572e-08 true resid norm 1.570197988244e-07 ||r(i)||/||b|| 7.426961290224e-10
   44 SNES Function norm 2.114171338084e+02 
      0 KSP preconditioned resid norm 4.636713819556e+03 true resid norm 2.114171338084e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.248004040828e-09 true resid norm 7.296845022145e-08 ||r(i)||/||b|| 3.451397193171e-10
   45 SNES Function norm 2.114170015592e+02 
      0 KSP preconditioned resid norm 6.321621979244e+03 true resid norm 2.114170015592e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.674310076597e-08 true resid norm 1.788726595879e-07 ||r(i)||/||b|| 8.460656345928e-10
   46 SNES Function norm 2.114164190412e+02 
      0 KSP preconditioned resid norm 7.414534654352e+03 true resid norm 2.114164190412e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.520165386851e-08 true resid norm 4.326674448851e-07 ||r(i)||/||b|| 2.046517705897e-09
   47 SNES Function norm 2.114162562951e+02 
      0 KSP preconditioned resid norm 9.810108530473e+03 true resid norm 2.114162562951e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.690981568660e-08 true resid norm 1.826808079437e-07 ||r(i)||/||b|| 8.640811787375e-10
   48 SNES Function norm 2.114160137794e+02 
      0 KSP preconditioned resid norm 1.136099973228e+04 true resid norm 2.114160137794e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.015775310105e-08 true resid norm 1.261980904678e-07 ||r(i)||/||b|| 5.969183138583e-10
   49 SNES Function norm 2.114158331092e+02 
      0 KSP preconditioned resid norm 1.315560026827e+04 true resid norm 2.114158331092e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 5.131165057283e-08 true resid norm 3.097361970470e-07 ||r(i)||/||b|| 1.465056767470e-09
   50 SNES Function norm 2.114158223286e+02 
      0 KSP preconditioned resid norm 1.794717411653e+04 true resid norm 2.114158223286e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.070352092082e-07 true resid norm 6.224398071454e-07 ||r(i)||/||b|| 2.944149592446e-09
   51 SNES Function norm 2.114157500757e+02 
      0 KSP preconditioned resid norm 2.077631649185e+04 true resid norm 2.114157500757e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.358541701826e-08 true resid norm 5.604873460299e-07 ||r(i)||/||b|| 2.651114431301e-09
   52 SNES Function norm 2.114156961871e+02 
      0 KSP preconditioned resid norm 2.405048247398e+04 true resid norm 2.114156961871e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 5.439555573169e-08 true resid norm 3.896791261159e-07 ||r(i)||/||b|| 1.843189191455e-09
   53 SNES Function norm 2.114156592651e+02 
      0 KSP preconditioned resid norm 2.914886710376e+04 true resid norm 2.114156592651e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.436687722935e-07 true resid norm 1.370484810325e-06 ||r(i)||/||b|| 6.482418639609e-09
   54 SNES Function norm 2.114156319126e+02 
      0 KSP preconditioned resid norm 3.373835363786e+04 true resid norm 2.114156319126e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.028533154395e-07 true resid norm 6.668879459515e-07 ||r(i)||/||b|| 3.154392794508e-09
   55 SNES Function norm 2.114156115019e+02 
      0 KSP preconditioned resid norm 3.904963797200e+04 true resid norm 2.114156115019e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.986152192528e-08 true resid norm 4.508900491313e-07 ||r(i)||/||b|| 2.132718799374e-09
   56 SNES Function norm 2.114155964342e+02 
      0 KSP preconditioned resid norm 4.594596937760e+04 true resid norm 2.114155964342e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.567731897537e-08 true resid norm 5.054454305445e-07 ||r(i)||/||b|| 2.390766996709e-09
   57 SNES Function norm 2.114155854345e+02 
      0 KSP preconditioned resid norm 5.317614782539e+04 true resid norm 2.114155854345e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.460625544912e-07 true resid norm 9.171801951021e-07 ||r(i)||/||b|| 4.338280894556e-09
   58 SNES Function norm 2.114155777135e+02 
      0 KSP preconditioned resid norm 6.400017583847e+04 true resid norm 2.114155777135e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.093685259962e-08 true resid norm 6.506435082837e-07 ||r(i)||/||b|| 3.077557081273e-09
   59 SNES Function norm 2.114155720468e+02 
      0 KSP preconditioned resid norm 7.406807436961e+04 true resid norm 2.114155720468e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.571128588743e-07 true resid norm 1.516640238741e-06 ||r(i)||/||b|| 7.173739493538e-09
   60 SNES Function norm 2.114155680388e+02 
      0 KSP preconditioned resid norm 8.892466432276e+04 true resid norm 2.114155680388e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.335222631846e-08 true resid norm 8.161361513986e-07 ||r(i)||/||b|| 3.860340839464e-09
   61 SNES Function norm 2.114155651044e+02 
      0 KSP preconditioned resid norm 1.029106627628e+05 true resid norm 2.114155651044e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 5.244108037270e-07 true resid norm 3.009616926997e-06 ||r(i)||/||b|| 1.423555037450e-08
   62 SNES Function norm 2.114155629136e+02 
      0 KSP preconditioned resid norm 1.190953464375e+05 true resid norm 2.114155629136e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 5.108162336627e-08 true resid norm 9.193387836453e-07 ||r(i)||/||b|| 4.348491525295e-09
   63 SNES Function norm 2.114155612779e+02 
      0 KSP preconditioned resid norm 1.378243866677e+05 true resid norm 2.114155612779e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.138603793608e-07 true resid norm 2.482322226118e-06 ||r(i)||/||b|| 1.174143573497e-08
   64 SNES Function norm 2.114155602404e+02 
      0 KSP preconditioned resid norm 1.698559088929e+05 true resid norm 2.114155602404e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.608494949489e-08 true resid norm 1.125534374356e-06 ||r(i)||/||b|| 5.323801015765e-09
   65 SNES Function norm 2.114155594364e+02 
      0 KSP preconditioned resid norm 1.965648734846e+05 true resid norm 2.114155594364e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.689853079655e-07 true resid norm 2.539748843840e-06 ||r(i)||/||b|| 1.201306493529e-08
   66 SNES Function norm 2.114155588361e+02 
      0 KSP preconditioned resid norm 2.274726625349e+05 true resid norm 2.114155588361e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.937782777025e-07 true resid norm 5.126870538363e-06 ||r(i)||/||b|| 2.425020450996e-08
   67 SNES Function norm 2.114155583879e+02 
      0 KSP preconditioned resid norm 2.632394448333e+05 true resid norm 2.114155583879e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.229084794159e-07 true resid norm 3.026195587359e-06 ||r(i)||/||b|| 1.431396823599e-08
   68 SNES Function norm 2.114155581765e+02 
      0 KSP preconditioned resid norm 3.367643572924e+05 true resid norm 2.114155581765e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.632794900620e-08 true resid norm 2.446819364813e-06 ||r(i)||/||b|| 1.157350663271e-08
   69 SNES Function norm 2.114155579720e+02 
      0 KSP preconditioned resid norm 3.897127303190e+05 true resid norm 2.114155579720e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.933486330809e-06 true resid norm 2.730183810468e-05 ||r(i)||/||b|| 1.291382638372e-07
   70 SNES Function norm 2.114155578193e+02 
      0 KSP preconditioned resid norm 4.509848215072e+05 true resid norm 2.114155578193e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.706389710731e-08 true resid norm 3.785217910694e-06 ||r(i)||/||b|| 1.790415970205e-08
   71 SNES Function norm 2.114155577054e+02 
      0 KSP preconditioned resid norm 5.242625554614e+05 true resid norm 2.114155577054e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.321749665453e-06 true resid norm 9.069981074375e-06 ||r(i)||/||b|| 4.290119976419e-08
   72 SNES Function norm 2.114155576210e+02 
      0 KSP preconditioned resid norm 6.066871925199e+05 true resid norm 2.114155576210e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.327204488687e-07 true resid norm 7.090997425424e-06 ||r(i)||/||b|| 3.354056581842e-08
   73 SNES Function norm 2.114155575580e+02 
      0 KSP preconditioned resid norm 7.020695777561e+05 true resid norm 2.114155575580e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.123816847857e-06 true resid norm 8.125899637337e-06 ||r(i)||/||b|| 3.843567489166e-08
   74 SNES Function norm 2.114155575109e+02 
      0 KSP preconditioned resid norm 8.124474305522e+05 true resid norm 2.114155575109e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.293550435673e-07 true resid norm 7.844829319667e-06 ||r(i)||/||b|| 3.710620643072e-08
   75 SNES Function norm 2.114155574865e+02 
      0 KSP preconditioned resid norm 1.029314742970e+06 true resid norm 2.114155574865e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.033392384072e-07 true resid norm 9.399925240813e-06 ||r(i)||/||b|| 4.446184260312e-08
   76 SNES Function norm 2.114155574646e+02 
      0 KSP preconditioned resid norm 1.191137352032e+06 true resid norm 2.114155574646e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.636526184597e-07 true resid norm 1.077765913224e-05 ||r(i)||/||b|| 5.097855267366e-08
   77 SNES Function norm 2.114155574575e+02 
      0 KSP preconditioned resid norm 1.563965951801e+06 true resid norm 2.114155574575e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.137216998536e-06 true resid norm 5.108848518118e-05 ||r(i)||/||b|| 2.416496013612e-07
   78 SNES Function norm 2.114155574558e+02 
      0 KSP preconditioned resid norm 2.108895892585e+06 true resid norm 2.114155574558e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.138935971807e-05 true resid norm 6.467637231732e-05 ||r(i)||/||b|| 3.059205911601e-07
   79 SNES Function norm 2.114155574511e+02 
      0 KSP preconditioned resid norm 2.564542138115e+06 true resid norm 2.114155574511e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.391129540980e-06 true resid norm 4.283623913226e-05 ||r(i)||/||b|| 2.026163052933e-07
   80 SNES Function norm 2.114155574475e+02 
      0 KSP preconditioned resid norm 2.967732587697e+06 true resid norm 2.114155574475e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.446508619620e-06 true resid norm 5.164837794317e-05 ||r(i)||/||b|| 2.442979058246e-07
   81 SNES Function norm 2.114155574449e+02 
      0 KSP preconditioned resid norm 3.434290246612e+06 true resid norm 2.114155574449e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.781979196562e-06 true resid norm 4.886329177979e-05 ||r(i)||/||b|| 2.311243901364e-07
   82 SNES Function norm 2.114155574429e+02 
      0 KSP preconditioned resid norm 3.974205574181e+06 true resid norm 2.114155574429e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.425678262963e-05 true resid norm 8.517034218173e-05 ||r(i)||/||b|| 4.028574964485e-07
   83 SNES Function norm 2.114155574416e+02 
      0 KSP preconditioned resid norm 4.785116089506e+06 true resid norm 2.114155574416e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.130667094145e-05 true resid norm 1.264608296603e-04 ||r(i)||/||b|| 5.981623641639e-07
   84 SNES Function norm 2.114155574406e+02 
      0 KSP preconditioned resid norm 5.537350742612e+06 true resid norm 2.114155574406e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.299149833543e-06 true resid norm 5.419308321331e-05 ||r(i)||/||b|| 2.563344148812e-07
   85 SNES Function norm 2.114155574399e+02 
      0 KSP preconditioned resid norm 6.767161611151e+06 true resid norm 2.114155574399e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.781986133631e-05 true resid norm 1.232616969009e-04 ||r(i)||/||b|| 5.830303994346e-07
   86 SNES Function norm 2.114155574394e+02 
      0 KSP preconditioned resid norm 7.867884017146e+06 true resid norm 2.114155574394e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.002696664504e-05 true resid norm 1.398533205346e-04 ||r(i)||/||b|| 6.615091255747e-07
   87 SNES Function norm 2.114155574390e+02 
      0 KSP preconditioned resid norm 9.102289060164e+06 true resid norm 2.114155574390e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.231201322143e-05 true resid norm 1.368407286352e-04 ||r(i)||/||b|| 6.472595030036e-07
   88 SNES Function norm 2.114155574387e+02 
      0 KSP preconditioned resid norm 1.063720954841e+07 true resid norm 2.114155574387e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.517853677131e-05 true resid norm 1.273061147947e-04 ||r(i)||/||b|| 6.021605805032e-07
   89 SNES Function norm 2.114155574385e+02 
      0 KSP preconditioned resid norm 1.230866787650e+07 true resid norm 2.114155574385e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.745229646688e-05 true resid norm 1.389443348166e-04 ||r(i)||/||b|| 6.572096041560e-07
   90 SNES Function norm 2.114155574384e+02 
      0 KSP preconditioned resid norm 1.436772950622e+07 true resid norm 2.114155574384e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.955301916175e-05 true resid norm 5.633754040143e-04 ||r(i)||/||b|| 2.664777421494e-06
   91 SNES Function norm 2.114155574383e+02 
      0 KSP preconditioned resid norm 1.662010558966e+07 true resid norm 2.114155574383e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.147116976734e-05 true resid norm 1.351152564949e-04 ||r(i)||/||b|| 6.390979837628e-07
   92 SNES Function norm 2.114155574382e+02 
      0 KSP preconditioned resid norm 1.923255924422e+07 true resid norm 2.114155574382e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.030030646485e-05 true resid norm 1.877965090103e-04 ||r(i)||/||b|| 8.882814078868e-07
   93 SNES Function norm 2.114155574381e+02 
      0 KSP preconditioned resid norm 2.226793244995e+07 true resid norm 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.411985876631e-04 true resid norm 1.342893408929e-03 ||r(i)||/||b|| 6.351913857247e-06
   94 SNES Function norm 2.114155574381e+02 
      0 KSP preconditioned resid norm 2.576953439486e+07 true resid norm 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.147595096922e-05 true resid norm 4.578229507611e-04 ||r(i)||/||b|| 2.165512114193e-06
   95 SNES Function norm 2.114155574380e+02 
      0 KSP preconditioned resid norm 2.981662207354e+07 true resid norm 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.456246888386e-06 true resid norm 2.856742906943e-04 ||r(i)||/||b|| 1.351245358460e-06
   96 SNES Function norm 2.114155574380e+02 
      0 KSP preconditioned resid norm 3.451395888978e+07 true resid norm 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.972976681165e-05 true resid norm 3.126288732572e-04 ||r(i)||/||b|| 1.478741096662e-06
  Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 96

-gideon

> On Sep 8, 2015, at 10:31 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Sep 8, 2015 at 9:19 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> Ok, going back and running the problem with the flags, -snes_fd -snes_mf_operator -ksp_monitor_true_residual -snes_max_funcs 500000, during the grid sequence, I still get:
> 
> Lets try using -pc_type lu instead of MUMPS, just to make sure. If we still
> get this, then the problem is really horribly conditioned without being singular.
> We are losing all precision for some reason. Is it possible that the two halves
> are so differently scaled that we make no progress on one with LU?
> 
>   Thanks,
> 
>      Matt
>  
>       0 SNES Function norm 9.131726454160e-09 
>         0 KSP preconditioned resid norm 4.201012306235e-09 true resid norm 9.131726454160e-09 ||r(i)||/||b|| 1.000000000000e+00
>         1 KSP preconditioned resid norm 4.811278613811e-14 true resid norm 7.117495013042e-13 ||r(i)||/||b|| 7.794249037978e-05
>         2 KSP preconditioned resid norm 1.729322442321e-19 true resid norm 2.611605096203e-13 ||r(i)||/||b|| 2.859924800982e-05
>       1 SNES Function norm 2.066497600915e-12 
>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>     0 SNES Function norm 5.066222213176e+03 
>       0 KSP preconditioned resid norm 3.135051257976e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.425321207844e-04 true resid norm 1.654961474093e+00 ||r(i)||/||b|| 3.266657885218e-04
>       2 KSP preconditioned resid norm 2.739130995469e-09 true resid norm 7.943827483388e-03 ||r(i)||/||b|| 1.567998234015e-06
>     1 SNES Function norm 8.487402654330e+02 
>       0 KSP preconditioned resid norm 2.670376525819e+02 true resid norm 8.487402654330e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.518628784729e-02 true resid norm 1.820466086409e+00 ||r(i)||/||b|| 2.144903641964e-03
>       2 KSP preconditioned resid norm 1.680654269556e-07 true resid norm 1.635299027240e-01 ||r(i)||/||b|| 1.926736710678e-04
>     2 SNES Function norm 6.547660268557e+02 
>       0 KSP preconditioned resid norm 1.529984375428e+02 true resid norm 6.547660268557e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.106652647444e-02 true resid norm 9.350438798847e-01 ||r(i)||/||b|| 1.428058026124e-03
>       2 KSP preconditioned resid norm 1.739404692247e-07 true resid norm 6.642022841611e-02 ||r(i)||/||b|| 1.014411647701e-04
>     3 SNES Function norm 5.771035131885e+02 
>       0 KSP preconditioned resid norm 1.360697738776e+02 true resid norm 5.771035131885e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.323138791370e-03 true resid norm 6.419077863137e-02 ||r(i)||/||b|| 1.112292286642e-04
>     4 SNES Function norm 5.238679723156e+02 
>       0 KSP preconditioned resid norm 1.528156345475e+02 true resid norm 5.238679723156e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.115694692015e-02 true resid norm 1.230231531838e+00 ||r(i)||/||b|| 2.348361795054e-03
>       2 KSP preconditioned resid norm 5.444815701049e-08 true resid norm 6.640979326953e-02 ||r(i)||/||b|| 1.267681873660e-04
>     5 SNES Function norm 4.755828064476e+02 
>       0 KSP preconditioned resid norm 1.872910183528e+02 true resid norm 4.755828064476e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.182180755863e-02 true resid norm 5.918459188490e-01 ||r(i)||/||b|| 1.244464498769e-03
>       2 KSP preconditioned resid norm 7.426717097996e-08 true resid norm 8.473303784961e-02 ||r(i)||/||b|| 1.781667392111e-04
>     6 SNES Function norm 4.224257767287e+02 
>       0 KSP preconditioned resid norm 2.578229460575e+02 true resid norm 4.224257767287e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.258348180527e-02 true resid norm 3.257118054248e-01 ||r(i)||/||b|| 7.710509712432e-04
>       2 KSP preconditioned resid norm 1.122972268815e-07 true resid norm 1.274729638407e-01 ||r(i)||/||b|| 3.017641698568e-04
>     7 SNES Function norm 3.811223987907e+02 
>       0 KSP preconditioned resid norm 5.218684581634e+02 true resid norm 3.811223987907e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.089005298384e-01 true resid norm 5.194667956652e+00 ||r(i)||/||b|| 1.362992039601e-02
>       2 KSP preconditioned resid norm 7.570453480881e-07 true resid norm 2.863356359719e-01 ||r(i)||/||b|| 7.512957435207e-04
>     8 SNES Function norm 3.771290745624e+02 
>       0 KSP preconditioned resid norm 5.827455360446e+03 true resid norm 3.771290745624e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.620225652579e-01 true resid norm 1.332007773441e+01 ||r(i)||/||b|| 3.531967867995e-02
>       2 KSP preconditioned resid norm 3.596801636560e-06 true resid norm 3.425465452374e+00 ||r(i)||/||b|| 9.083005483862e-03
>     9 SNES Function norm 3.767897776540e+02 
>       0 KSP preconditioned resid norm 5.185115189077e+03 true resid norm 3.767897776540e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.021105455600e-01 true resid norm 1.043813038739e+01 ||r(i)||/||b|| 2.770279611187e-02
>       2 KSP preconditioned resid norm 6.951879108628e-06 true resid norm 2.816562934830e+00 ||r(i)||/||b|| 7.475157506572e-03
>    10 SNES Function norm 3.746968158051e+02 
>       0 KSP preconditioned resid norm 8.165048723911e+02 true resid norm 3.746968158051e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.177528145230e-01 true resid norm 2.394044923481e+00 ||r(i)||/||b|| 6.389285476945e-03
>       2 KSP preconditioned resid norm 7.325770200602e-07 true resid norm 4.877299135281e-01 ||r(i)||/||b|| 1.301665487816e-03
>    11 SNES Function norm 3.726877791108e+02 
>       0 KSP preconditioned resid norm 5.024547551936e+02 true resid norm 3.726877791108e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.640017871568e-01 true resid norm 1.648101091752e+00 ||r(i)||/||b|| 4.422203206351e-03
>       2 KSP preconditioned resid norm 1.005738633834e-06 true resid norm 2.935508517682e-01 ||r(i)||/||b|| 7.876589151074e-04
>    12 SNES Function norm 3.668832918639e+02 
>       0 KSP preconditioned resid norm 7.762104953873e+02 true resid norm 3.668832918639e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.912259329681e-02 true resid norm 1.334600681831e+00 ||r(i)||/||b|| 3.637670919956e-03
>       2 KSP preconditioned resid norm 7.640320635302e-07 true resid norm 4.471334143387e-01 ||r(i)||/||b|| 1.218734742776e-03
>    13 SNES Function norm 3.631997889667e+02 
>       0 KSP preconditioned resid norm 2.515761606628e+02 true resid norm 3.631997889667e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.648297544435e-01 true resid norm 2.866637955219e+00 ||r(i)||/||b|| 7.892730233611e-03
>       2 KSP preconditioned resid norm 7.305731711999e-07 true resid norm 1.472589913546e-01 ||r(i)||/||b|| 4.054490003243e-04
>    14 SNES Function norm 3.557844076846e+02 
>       0 KSP preconditioned resid norm 6.995145168850e+02 true resid norm 3.557844076846e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.626527197886e-02 true resid norm 5.642316830909e-01 ||r(i)||/||b|| 1.585880861848e-03
>       2 KSP preconditioned resid norm 1.040702263139e-06 true resid norm 4.073703976052e-01 ||r(i)||/||b|| 1.144992272866e-03
>    15 SNES Function norm 3.534147071188e+02 
>       0 KSP preconditioned resid norm 2.698507093385e+02 true resid norm 3.534147071188e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.592052963868e-01 true resid norm 1.332346399928e+00 ||r(i)||/||b|| 3.769923472595e-03
>       2 KSP preconditioned resid norm 5.021635239169e-07 true resid norm 1.573071613558e-01 ||r(i)||/||b|| 4.451064378114e-04
>    16 SNES Function norm 3.499900270010e+02 
>       0 KSP preconditioned resid norm 7.295084253442e+01 true resid norm 3.499900270010e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.040248007780e-02 true resid norm 1.427506717563e-01 ||r(i)||/||b|| 4.078706841436e-04
>       2 KSP preconditioned resid norm 1.246491224109e-07 true resid norm 4.074099442269e-02 ||r(i)||/||b|| 1.164061581177e-04
>    17 SNES Function norm 3.473017914657e+02 
>       0 KSP preconditioned resid norm 4.169372990718e+02 true resid norm 3.473017914657e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.140552389441e-01 true resid norm 1.893314948416e+00 ||r(i)||/||b|| 5.451497789361e-03
>       2 KSP preconditioned resid norm 7.751532205539e-07 true resid norm 2.494992549211e-01 ||r(i)||/||b|| 7.183932275966e-04
>    18 SNES Function norm 3.464416771335e+02 
>       0 KSP preconditioned resid norm 2.968698304201e+02 true resid norm 3.464416771335e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.683113115222e-02 true resid norm 2.501212385243e-01 ||r(i)||/||b|| 7.219721385540e-04
>       2 KSP preconditioned resid norm 5.274081741342e-07 true resid norm 1.712806350336e-01 ||r(i)||/||b|| 4.943996243489e-04
>    19 SNES Function norm 3.460747441983e+02 
>       0 KSP preconditioned resid norm 3.707198448214e+02 true resid norm 3.460747441983e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.740664131652e-02 true resid norm 9.477346570578e-01 ||r(i)||/||b|| 2.738525919461e-03
>       2 KSP preconditioned resid norm 4.649501179917e-07 true resid norm 2.217443031583e-01 ||r(i)||/||b|| 6.407410736429e-04
>    20 SNES Function norm 3.455596398964e+02 
>       0 KSP preconditioned resid norm 5.000331867417e+02 true resid norm 3.455596398964e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.821124737573e-02 true resid norm 7.651194558377e-01 ||r(i)||/||b|| 2.214145888296e-03
>       2 KSP preconditioned resid norm 5.030283955029e-07 true resid norm 2.918161448112e-01 ||r(i)||/||b|| 8.444740389784e-04
>    21 SNES Function norm 3.453222949611e+02 
>       0 KSP preconditioned resid norm 7.399233760260e+02 true resid norm 3.453222949611e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.030913689264e-01 true resid norm 8.930798988128e-01 ||r(i)||/||b|| 2.586221370136e-03
>       2 KSP preconditioned resid norm 1.497261432173e-06 true resid norm 4.385255355661e-01 ||r(i)||/||b|| 1.269902181136e-03
>    22 SNES Function norm 3.452850731057e+02 
>       0 KSP preconditioned resid norm 1.850020325701e+03 true resid norm 3.452850731057e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.670597549481e-01 true resid norm 6.205758169419e+00 ||r(i)||/||b|| 1.797285389027e-02
>       2 KSP preconditioned resid norm 4.122440586032e-06 true resid norm 1.109676174925e+00 ||r(i)||/||b|| 3.213797124053e-03
>    23 SNES Function norm 3.452722963704e+02 
>       0 KSP preconditioned resid norm 3.362446711623e+03 true resid norm 3.452722963704e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.181412468129e+00 true resid norm 1.564006484993e+01 ||r(i)||/||b|| 4.529776936738e-02
>       2 KSP preconditioned resid norm 1.165193864469e-05 true resid norm 1.869411988347e+00 ||r(i)||/||b|| 5.414312147250e-03
>    24 SNES Function norm 3.452679622612e+02 
>       0 KSP preconditioned resid norm 6.149140486077e+03 true resid norm 3.452679622612e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.400761945257e-01 true resid norm 3.795886928810e+00 ||r(i)||/||b|| 1.099403171945e-02
>       2 KSP preconditioned resid norm 1.754461070240e-06 true resid norm 3.648059240080e+00 ||r(i)||/||b|| 1.056587821294e-02
>    25 SNES Function norm 3.452667776779e+02 
>       0 KSP preconditioned resid norm 1.101683177789e+04 true resid norm 3.452667776779e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.317993935110e+00 true resid norm 2.454711939414e+01 ||r(i)||/||b|| 7.109609432809e-02
>       2 KSP preconditioned resid norm 3.457477291813e-05 true resid norm 6.807469339003e+00 ||r(i)||/||b|| 1.971654899665e-02
>    26 SNES Function norm 3.452666586474e+02 
>       0 KSP preconditioned resid norm 3.879190850784e+04 true resid norm 3.452666586474e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.739562350939e+01 true resid norm 1.221587956319e+02 ||r(i)||/||b|| 3.538099974971e-01
>       2 KSP preconditioned resid norm 7.455463420492e-05 true resid norm 3.080916578781e+01 ||r(i)||/||b|| 8.923295955800e-02
>    27 SNES Function norm 3.452666425264e+02 
>       0 KSP preconditioned resid norm 5.709318799171e+04 true resid norm 3.452666425264e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.292899448408e-01 true resid norm 2.801354737138e+01 ||r(i)||/||b|| 8.113597990933e-02
>    28 SNES Function norm 3.452666374187e+02 
>       0 KSP preconditioned resid norm 2.488913348294e+05 true resid norm 3.452666374187e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.691998534170e+02 true resid norm 7.244072115265e+03 ||r(i)||/||b|| 2.098109498625e+01
>       2 KSP preconditioned resid norm 2.503929513070e-03 true resid norm 2.928791518636e+03 ||r(i)||/||b|| 8.482694825461e+00
>    29 SNES Function norm 3.452666374177e+02 
>       0 KSP preconditioned resid norm 2.533150641403e+05 true resid norm 3.452666374177e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.817060595281e+02 true resid norm 1.333479168158e+03 ||r(i)||/||b|| 3.862172082803e+00
>       2 KSP preconditioned resid norm 8.930133492283e-04 true resid norm 5.412764197416e+02 ||r(i)||/||b|| 1.567705538507e+00
>    30 SNES Function norm 3.452666373128e+02 
>       0 KSP preconditioned resid norm 2.169409532487e+05 true resid norm 3.452666373128e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.325438416139e+02 true resid norm 9.441618139691e+02 ||r(i)||/||b|| 2.734587451940e+00
>       2 KSP preconditioned resid norm 8.796591247860e-04 true resid norm 2.978002325996e+02 ||r(i)||/||b|| 8.625224693509e-01
>    31 SNES Function norm 3.452666372449e+02 
>       0 KSP preconditioned resid norm 1.459475274582e+05 true resid norm 3.452666372449e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.978987062939e+02 true resid norm 1.411576222768e+03 ||r(i)||/||b|| 4.088365542735e+00
>       2 KSP preconditioned resid norm 1.235854958341e-03 true resid norm 3.579390387115e+02 ||r(i)||/||b|| 1.036703231936e+00
>    32 SNES Function norm 3.452666369928e+02 
>       0 KSP preconditioned resid norm 1.264234515603e+05 true resid norm 3.452666369928e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.676975931928e+01 true resid norm 1.855746501202e+02 ||r(i)||/||b|| 5.374821376792e-01
>       2 KSP preconditioned resid norm 9.286140031498e-05 true resid norm 3.495715551553e+01 ||r(i)||/||b|| 1.012468387331e-01
>    33 SNES Function norm 3.452666245901e+02 
>       0 KSP preconditioned resid norm 1.544591438108e+06 true resid norm 3.452666245901e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.501087626485e+01 true resid norm 3.233291920369e+02 ||r(i)||/||b|| 9.364623424597e-01
>       2 KSP preconditioned resid norm 1.867215855867e-04 true resid norm 7.114686634655e+01 ||r(i)||/||b|| 2.060635499623e-01
>    34 SNES Function norm 3.452666216061e+02 
>       0 KSP preconditioned resid norm 3.431570281248e+05 true resid norm 3.452666216061e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.140176766342e+02 true resid norm 1.760366951993e+03 ||r(i)||/||b|| 5.098572644538e+00
>       2 KSP preconditioned resid norm 2.324744486242e-03 true resid norm 1.080605579953e+03 ||r(i)||/||b|| 3.129771348664e+00
>    35 SNES Function norm 3.452666215921e+02 
>       0 KSP preconditioned resid norm 3.734063885343e+05 true resid norm 3.452666215921e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.279219133674e+00 true resid norm 1.691481367261e+02 ||r(i)||/||b|| 4.899058470990e-01
>       2 KSP preconditioned resid norm 4.046466703016e-04 true resid norm 1.964786369469e+02 ||r(i)||/||b|| 5.690635139908e-01
>    36 SNES Function norm 3.452666212200e+02 
>       0 KSP preconditioned resid norm 7.952390829283e+05 true resid norm 3.452666212200e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.707607129733e+01 true resid norm 2.806640653978e+02 ||r(i)||/||b|| 8.128908158167e-01
>       2 KSP preconditioned resid norm 1.296391885225e-04 true resid norm 1.079623998255e+02 ||r(i)||/||b|| 3.126928384910e-01
>    37 SNES Function norm 3.452666199636e+02 
>       0 KSP preconditioned resid norm 9.358626316710e+05 true resid norm 3.452666199636e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.722696571742e+01 true resid norm 3.251492717238e+02 ||r(i)||/||b|| 9.417338744130e-01
>       2 KSP preconditioned resid norm 2.862108374145e-04 true resid norm 3.320104402606e+01 ||r(i)||/||b|| 9.616059620694e-02
>    38 SNES Function norm 3.452666133867e+02 
>       0 KSP preconditioned resid norm 8.365871848022e+04 true resid norm 3.452666133867e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.482543031390e+02 true resid norm 1.772891732407e+03 ||r(i)||/||b|| 5.134848443690e+00
>       2 KSP preconditioned resid norm 1.439398367387e-03 true resid norm 3.088902452162e+02 ||r(i)||/||b|| 8.946426710257e-01
>    39 SNES Function norm 3.452666132332e+02 
>       0 KSP preconditioned resid norm 8.960112658041e+04 true resid norm 3.452666132332e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.505891399150e+01 true resid norm 1.787713293837e+02 ||r(i)||/||b|| 5.177776319280e-01
>       2 KSP preconditioned resid norm 6.384856870578e-05 true resid norm 2.543851208517e+01 ||r(i)||/||b|| 7.367787996341e-02
>    40 SNES Function norm 3.452665904125e+02 
>       0 KSP preconditioned resid norm 7.796808144307e+05 true resid norm 3.452665904125e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.020764189760e+01 true resid norm 6.961545395677e+02 ||r(i)||/||b|| 2.016281212544e+00
>       2 KSP preconditioned resid norm 3.506849883654e-04 true resid norm 4.956460306859e+02 ||r(i)||/||b|| 1.435545878024e+00
>    41 SNES Function norm 3.452665902941e+02 
>       0 KSP preconditioned resid norm 4.990797566798e+05 true resid norm 3.452665902941e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.012324354359e+02 true resid norm 7.390530765626e+02 ||r(i)||/||b|| 2.140528789458e+00
>       2 KSP preconditioned resid norm 7.111862680292e-04 true resid norm 4.066260265930e+02 ||r(i)||/||b|| 1.177716112777e+00
>    42 SNES Function norm 3.452665901159e+02 
>       0 KSP preconditioned resid norm 3.472654025442e+05 true resid norm 3.452665901159e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.552680791160e+02 true resid norm 1.064105325830e+03 ||r(i)||/||b|| 3.081981739016e+00
>       2 KSP preconditioned resid norm 2.695346315896e-03 true resid norm 5.359619569435e+02 ||r(i)||/||b|| 1.552313407340e+00
>    43 SNES Function norm 3.452665900083e+02 
>       0 KSP preconditioned resid norm 2.823873507449e+05 true resid norm 3.452665900083e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.616335635707e+00 true resid norm 1.564312339233e+02 ||r(i)||/||b|| 4.530737651723e-01
>       2 KSP preconditioned resid norm 1.580836053583e-04 true resid norm 1.777870114826e+02 ||r(i)||/||b|| 5.149267743465e-01
>    44 SNES Function norm 3.452665895210e+02 
>       0 KSP preconditioned resid norm 4.764215602672e+05 true resid norm 3.452665895210e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.491949190147e+01 true resid norm 4.652947485976e+02 ||r(i)||/||b|| 1.347639078670e+00
>       2 KSP preconditioned resid norm 2.266106888017e-04 true resid norm 1.316722015429e+02 ||r(i)||/||b|| 3.813638664707e-01
>    45 SNES Function norm 3.452665889075e+02 
>       0 KSP preconditioned resid norm 1.439659890301e+05 true resid norm 3.452665889075e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.431461947162e+01 true resid norm 1.403614551576e+02 ||r(i)||/||b|| 4.065306625865e-01
>       2 KSP preconditioned resid norm 4.711117728059e-04 true resid norm 1.095117712535e+02 ||r(i)||/||b|| 3.171803318704e-01
>    46 SNES Function norm 3.452665876547e+02 
>       0 KSP preconditioned resid norm 2.157313334766e+05 true resid norm 3.452665876547e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.967750174401e+00 true resid norm 1.053591658730e+02 ||r(i)||/||b|| 3.051530893525e-01
>       2 KSP preconditioned resid norm 4.512059841816e-04 true resid norm 1.189061020519e+02 ||r(i)||/||b|| 3.443892525473e-01
>    47 SNES Function norm 3.452665866229e+02 
>       0 KSP preconditioned resid norm 3.981118078251e+05 true resid norm 3.452665866229e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.013119377232e+01 true resid norm 4.969789311062e+02 ||r(i)||/||b|| 1.439406390196e+00
>       2 KSP preconditioned resid norm 1.872910444347e-04 true resid norm 1.396396719776e+02 ||r(i)||/||b|| 4.044401554852e-01
>    48 SNES Function norm 3.452665850954e+02 
>       0 KSP preconditioned resid norm 1.967554630062e+05 true resid norm 3.452665850954e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.006970327024e+00 true resid norm 9.616098906643e+01 ||r(i)||/||b|| 2.785122951874e-01
>       2 KSP preconditioned resid norm 3.184368801558e-05 true resid norm 9.951979756268e+01 ||r(i)||/||b|| 2.882404549377e-01
>    49 SNES Function norm 3.452665835600e+02 
>       0 KSP preconditioned resid norm 3.965648247310e+05 true resid norm 3.452665835600e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.862969730098e+01 true resid norm 1.790616593203e+02 ||r(i)||/||b|| 5.186185627174e-01
>       2 KSP preconditioned resid norm 2.964457441143e-04 true resid norm 1.447818294544e+02 ||r(i)||/||b|| 4.193334552146e-01
>    50 SNES Function norm 3.452665828612e+02 
>       0 KSP preconditioned resid norm 1.265022286683e+06 true resid norm 3.452665828612e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.401066920177e+01 true resid norm 3.143671803790e+02 ||r(i)||/||b|| 9.105056671685e-01
>       2 KSP preconditioned resid norm 1.444898193175e-04 true resid norm 7.578180059616e+01 ||r(i)||/||b|| 2.194877939480e-01
>    51 SNES Function norm 3.452665817118e+02 
>       0 KSP preconditioned resid norm 2.063334812849e+05 true resid norm 3.452665817118e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.074628697499e+01 true resid norm 1.155569679685e+02 ||r(i)||/||b|| 3.346891187543e-01
>       2 KSP preconditioned resid norm 1.326420665890e-04 true resid norm 9.261090091148e+01 ||r(i)||/||b|| 2.682301323584e-01
>    52 SNES Function norm 3.452665807811e+02 
>       0 KSP preconditioned resid norm 6.549963238965e+06 true resid norm 3.452665807811e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.246352248726e+01 true resid norm 3.604213542889e+02 ||r(i)||/||b|| 1.043892963731e+00
>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 52
> 
> which is a little different, but not so much different, than what i get if I use my Jacobian.
> 
> -gideon
> 
>> On Sep 8, 2015, at 10:00 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>> 
>> Yup, now it?s working.
>> -gideon
>> 
>>> On Sep 8, 2015, at 9:58 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>> 
>>> On Tue, Sep 8, 2015 at 8:51 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>> Going back to this point, a case that previously worked, is now giving me problems that I don?t understand:
>>> 
>>>   0 SNES Function norm 1.421454390131e-02 
>>>     0 KSP preconditioned resid norm 7.635339584553e-03 true resid norm 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP preconditioned resid norm 2.838956718317e-07 true resid norm 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
>>>     2 KSP preconditioned resid norm 5.889748199467e-12 true resid norm 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
>>>   1 SNES Function norm 4.476139939604e-06 
>>>     0 KSP preconditioned resid norm 4.897102387844e-05 true resid norm 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP preconditioned resid norm 1.133649519960e-09 true resid norm 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
>>>     2 KSP preconditioned resid norm 2.223328042462e-14 true resid norm 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
>>>   2 SNES Function norm 5.447761863330e-09 
>>>     0 KSP preconditioned resid norm 9.703890346186e-09 true resid norm 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP preconditioned resid norm 1.982949185218e-13 true resid norm 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
>>>     2 KSP preconditioned resid norm 7.045835741315e-18 true resid norm 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
>>> Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT iterations 2
>>> 
>>> Somehow the maximum number of function evaluations is being exceeded. Oh, are you using -snes_fd? This uses a function evaluation
>>> for each column, so you might have to up it > 10000. The convergence is fine.
>>> 
>>>   Thanks,
>>> 
>>>      Matt
>>>  
>>> -gideon
>>> 
>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>> 
>>>> 
>>>>   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>>> 
>>>>  Barry
>>>> 
>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>> 
>>>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>>> 
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>>> 
>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>>>> 
>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>>> 
>>>>>>   Matt
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>>> 
>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>>> 
>>>>>>> Gideon,
>>>>>>> 
>>>>>>>  In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>>> 
>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>>> 
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>>>> 
>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>>> 
>>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>>> 
>>>>>>>>>>   0 SNES Function norm 3.857327250267e-09 
>>>>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>>   1 SNES Function norm 2.479697868412e-12 
>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>>> 
>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>>> 
>>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>>> 
>>>>>>>>>>>>  0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>>>  1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>>> 
>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> then 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>>>    type: seqaij
>>>>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>>      MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>>>        PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>>>          [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>>>          [0]  143924 
>>>>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>>>          [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>>>          [0] 15991 
>>>>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>>    SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>>    etc
>>>>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/53a53673/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Sep  8 21:53:25 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 Sep 2015 21:53:25 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
Message-ID: <40AC7B77-D631-4578-9AE6-2D4180970ABF@mcs.anl.gov>


  Is this with -snes_fd?

> On Sep 8, 2015, at 9:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> Your comment about the different pieces scaling different made me think of something, and I made a slight modification.  It still fails, but the conditioning looks much better, no?
> 
> 
>       0 SNES Function norm 4.027474333769e-09 
>         0 KSP preconditioned resid norm 9.390682304142e-09 true resid norm 4.027474333769e-09 ||r(i)||/||b|| 1.000000000000e+00
>         1 KSP preconditioned resid norm 4.937459249116e-20 true resid norm 3.737772204700e-19 ||r(i)||/||b|| 9.280685350023e-11
>       1 SNES Function norm 2.666655509899e-12 
>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>     0 SNES Function norm 5.066222213175e+03 
>       0 KSP preconditioned resid norm 3.135087050015e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.220576053756e-11 true resid norm 1.206309587478e-07 ||r(i)||/||b|| 2.381083056997e-11
>     1 SNES Function norm 8.484697184819e+02 
>       0 KSP preconditioned resid norm 2.668083368240e+02 true resid norm 8.484697184819e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.388166746665e-10 true resid norm 2.735732864000e-08 ||r(i)||/||b|| 3.224314085004e-11
>     2 SNES Function norm 6.549559723821e+02 
>       0 KSP preconditioned resid norm 1.552154273063e+02 true resid norm 6.549559723821e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.740227244709e-10 true resid norm 1.800722435354e-08 ||r(i)||/||b|| 2.749379364852e-11
>     3 SNES Function norm 5.770723278635e+02 
>       0 KSP preconditioned resid norm 1.348596653013e+02 true resid norm 5.770723278635e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.142365551700e-10 true resid norm 7.387664544933e-09 ||r(i)||/||b|| 1.280197331985e-11
>     4 SNES Function norm 5.237702241066e+02 
>       0 KSP preconditioned resid norm 1.524401223470e+02 true resid norm 5.237702241066e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.168592748079e-10 true resid norm 9.512653260875e-09 ||r(i)||/||b|| 1.816188248788e-11
>     5 SNES Function norm 4.753909020302e+02 
>       0 KSP preconditioned resid norm 1.864261890160e+02 true resid norm 4.753909020302e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.744912816503e-10 true resid norm 1.889384889937e-08 ||r(i)||/||b|| 3.974381675939e-11
>     6 SNES Function norm 4.221784591168e+02 
>       0 KSP preconditioned resid norm 2.550992140633e+02 true resid norm 4.221784591168e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.697718851690e-10 true resid norm 1.666255394677e-08 ||r(i)||/||b|| 3.946803439861e-11
>     7 SNES Function norm 3.806525080909e+02 
>       0 KSP preconditioned resid norm 5.183605288372e+02 true resid norm 3.806525080909e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.240268561213e-09 true resid norm 1.041310587019e-07 ||r(i)||/||b|| 2.735593657958e-10
>     8 SNES Function norm 3.762054656363e+02 
>       0 KSP preconditioned resid norm 5.320776249886e+03 true resid norm 3.762054656363e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.328856456273e-09 true resid norm 7.860313076537e-08 ||r(i)||/||b|| 2.089367059897e-10
>     9 SNES Function norm 3.758975227207e+02 
>       0 KSP preconditioned resid norm 6.252711400154e+03 true resid norm 3.758975227207e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.900963654627e-09 true resid norm 1.347299589292e-07 ||r(i)||/||b|| 3.584220453330e-10
>    10 SNES Function norm 3.757032042008e+02 
>       0 KSP preconditioned resid norm 5.890497230519e+02 true resid norm 3.757032042008e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.497822807230e-09 true resid norm 3.605311757144e-08 ||r(i)||/||b|| 9.596169840535e-11
>    11 SNES Function norm 3.728798164183e+02 
>       0 KSP preconditioned resid norm 1.081356468099e+03 true resid norm 3.728798164183e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.713971332514e-09 true resid norm 3.705072372867e-08 ||r(i)||/||b|| 9.936371478768e-11
>    12 SNES Function norm 3.723078736486e+02 
>       0 KSP preconditioned resid norm 3.745123875883e+03 true resid norm 3.723078736486e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.885886422519e-08 true resid norm 2.837206039485e-07 ||r(i)||/||b|| 7.620591022374e-10
>    13 SNES Function norm 3.721848051981e+02 
>       0 KSP preconditioned resid norm 3.495480420016e+03 true resid norm 3.721848051981e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.874759607493e-08 true resid norm 2.751735241072e-07 ||r(i)||/||b|| 7.393464759014e-10
>    14 SNES Function norm 3.720227571463e+02 
>       0 KSP preconditioned resid norm 1.265998097646e+04 true resid norm 3.720227571463e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.125038941589e-08 true resid norm 5.141611324046e-07 ||r(i)||/||b|| 1.382069033488e-09
>    15 SNES Function norm 3.720051993666e+02 
>       0 KSP preconditioned resid norm 2.912369751490e+03 true resid norm 3.720051993666e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.170907138699e-09 true resid norm 3.623872061163e-08 ||r(i)||/||b|| 9.741455407971e-11
>    16 SNES Function norm 3.718945615434e+02 
>       0 KSP preconditioned resid norm 1.082053028195e+03 true resid norm 3.718945615434e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.660982704773e-09 true resid norm 9.903133228018e-08 ||r(i)||/||b|| 2.662887348210e-10
>    17 SNES Function norm 3.700412140239e+02 
>       0 KSP preconditioned resid norm 3.676306553630e+01 true resid norm 3.700412140239e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.389666775474e-10 true resid norm 4.384613797840e-10 ||r(i)||/||b|| 1.184898771183e-12
>    18 SNES Function norm 3.351961239413e+02 
>       0 KSP preconditioned resid norm 1.071519127966e+02 true resid norm 3.351961239413e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.391049839274e-11 true resid norm 1.771280721778e-09 ||r(i)||/||b|| 5.284311468017e-12
>    19 SNES Function norm 3.096040290055e+02 
>       0 KSP preconditioned resid norm 3.362840348158e+02 true resid norm 3.096040290055e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.847034649159e-10 true resid norm 3.697317355249e-09 ||r(i)||/||b|| 1.194208411023e-11
>    20 SNES Function norm 3.008687462984e+02 
>       0 KSP preconditioned resid norm 7.161404453567e+01 true resid norm 3.008687462984e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.352839847013e-11 true resid norm 7.837384020140e-10 ||r(i)||/||b|| 2.604917963917e-12
>    21 SNES Function norm 2.752608806757e+02 
>       0 KSP preconditioned resid norm 1.727563200247e+02 true resid norm 2.752608806757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.663026698374e-10 true resid norm 5.807801532466e-09 ||r(i)||/||b|| 2.109926233692e-11
>    22 SNES Function norm 2.707559177016e+02 
>       0 KSP preconditioned resid norm 4.125046548626e+02 true resid norm 2.707559177016e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.109007703208e-10 true resid norm 5.875389803064e-09 ||r(i)||/||b|| 2.169994972940e-11
>    23 SNES Function norm 2.698855074973e+02 
>       0 KSP preconditioned resid norm 7.414116220947e+01 true resid norm 2.698855074973e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.222534270039e-11 true resid norm 6.928234778488e-10 ||r(i)||/||b|| 2.567101450810e-12
>    24 SNES Function norm 2.616892369202e+02 
>       0 KSP preconditioned resid norm 1.819161466540e+02 true resid norm 2.616892369202e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.013662651542e-09 true resid norm 4.195842561304e-09 ||r(i)||/||b|| 1.603368411588e-11
>    25 SNES Function norm 2.600248325147e+02 
>       0 KSP preconditioned resid norm 1.265123744997e+02 true resid norm 2.600248325147e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.572744639399e-09 true resid norm 7.386966838391e-09 ||r(i)||/||b|| 2.840869761150e-11
>    26 SNES Function norm 2.593065715720e+02 
>       0 KSP preconditioned resid norm 1.314965938655e+02 true resid norm 2.593065715720e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.156889834113e-10 true resid norm 3.157778817609e-09 ||r(i)||/||b|| 1.217778168314e-11
>    27 SNES Function norm 2.517348459439e+02 
>       0 KSP preconditioned resid norm 2.618005944700e+01 true resid norm 2.517348459439e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.483924363849e-11 true resid norm 7.418968110702e-10 ||r(i)||/||b|| 2.947135936976e-12
>    28 SNES Function norm 2.298635352641e+02 
>       0 KSP preconditioned resid norm 8.770042104744e+01 true resid norm 2.298635352641e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.098440763657e-09 true resid norm 3.936819814733e-09 ||r(i)||/||b|| 1.712676962968e-11
>    29 SNES Function norm 2.269871118080e+02 
>       0 KSP preconditioned resid norm 2.285655807501e+02 true resid norm 2.269871118080e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.216116049541e-08 true resid norm 9.327420966977e-08 ||r(i)||/||b|| 4.109229327024e-10
>    30 SNES Function norm 2.252105956612e+02 
>       0 KSP preconditioned resid norm 8.597156834231e+01 true resid norm 2.252105956612e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.431615405953e-09 true resid norm 4.297181487549e-09 ||r(i)||/||b|| 1.908072519826e-11
>    31 SNES Function norm 2.251883212710e+02 
>       0 KSP preconditioned resid norm 7.066983917118e+01 true resid norm 2.251883212710e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.998573955346e-10 true resid norm 3.454505271056e-09 ||r(i)||/||b|| 1.534051700176e-11
>    32 SNES Function norm 2.117895362999e+02 
>       0 KSP preconditioned resid norm 3.830985324064e+02 true resid norm 2.117895362999e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.016573825110e-09 true resid norm 1.942657883557e-08 ||r(i)||/||b|| 9.172586698552e-11
>    33 SNES Function norm 2.116404150058e+02 
>       0 KSP preconditioned resid norm 5.329232125733e+02 true resid norm 2.116404150058e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.609537995048e-09 true resid norm 1.451505196923e-08 ||r(i)||/||b|| 6.858355465253e-11
>    34 SNES Function norm 2.115566314991e+02 
>       0 KSP preconditioned resid norm 6.609724236205e+02 true resid norm 2.115566314991e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.254105497370e-10 true resid norm 1.178528765891e-08 ||r(i)||/||b|| 5.570748397436e-11
>    35 SNES Function norm 2.115028944726e+02 
>       0 KSP preconditioned resid norm 8.217040556045e+02 true resid norm 2.115028944726e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.803324342385e-09 true resid norm 2.957366914143e-08 ||r(i)||/||b|| 1.398263093050e-10
>    36 SNES Function norm 2.114773510911e+02 
>       0 KSP preconditioned resid norm 1.091957913212e+03 true resid norm 2.114773510911e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.264170340401e-08 true resid norm 1.635833495717e-07 ||r(i)||/||b|| 7.735265678696e-10
>    37 SNES Function norm 2.114573155596e+02 
>       0 KSP preconditioned resid norm 1.295967660499e+03 true resid norm 2.114573155596e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.787403648651e-08 true resid norm 1.913077973453e-07 ||r(i)||/||b|| 9.047111793648e-10
>    38 SNES Function norm 2.114434864633e+02 
>       0 KSP preconditioned resid norm 1.567323979910e+03 true resid norm 2.114434864633e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.869044911030e-09 true resid norm 3.425540409553e-08 ||r(i)||/||b|| 1.620073744928e-10
>    39 SNES Function norm 2.114337906450e+02 
>       0 KSP preconditioned resid norm 1.835099219771e+03 true resid norm 2.114337906450e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.224665285731e-10 true resid norm 1.690151657152e-08 ||r(i)||/||b|| 7.993763210677e-11
>    40 SNES Function norm 2.114267348757e+02 
>       0 KSP preconditioned resid norm 2.146249211809e+03 true resid norm 2.114267348757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.809299780773e-08 true resid norm 1.157600089098e-07 ||r(i)||/||b|| 5.475183116166e-10
>    41 SNES Function norm 2.114215910347e+02 
>       0 KSP preconditioned resid norm 2.510692990698e+03 true resid norm 2.114215910347e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.485943641401e-08 true resid norm 9.439839396991e-08 ||r(i)||/||b|| 4.464936315535e-10
>    42 SNES Function norm 2.114206676373e+02 
>       0 KSP preconditioned resid norm 3.401544811763e+03 true resid norm 2.114206676373e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.910138749741e-09 true resid norm 3.806926432018e-08 ||r(i)||/||b|| 1.800640625423e-10
>    43 SNES Function norm 2.114186309697e+02 
>       0 KSP preconditioned resid norm 3.955962524189e+03 true resid norm 2.114186309697e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.500201400572e-08 true resid norm 1.570197988244e-07 ||r(i)||/||b|| 7.426961290224e-10
>    44 SNES Function norm 2.114171338084e+02 
>       0 KSP preconditioned resid norm 4.636713819556e+03 true resid norm 2.114171338084e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.248004040828e-09 true resid norm 7.296845022145e-08 ||r(i)||/||b|| 3.451397193171e-10
>    45 SNES Function norm 2.114170015592e+02 
>       0 KSP preconditioned resid norm 6.321621979244e+03 true resid norm 2.114170015592e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.674310076597e-08 true resid norm 1.788726595879e-07 ||r(i)||/||b|| 8.460656345928e-10
>    46 SNES Function norm 2.114164190412e+02 
>       0 KSP preconditioned resid norm 7.414534654352e+03 true resid norm 2.114164190412e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.520165386851e-08 true resid norm 4.326674448851e-07 ||r(i)||/||b|| 2.046517705897e-09
>    47 SNES Function norm 2.114162562951e+02 
>       0 KSP preconditioned resid norm 9.810108530473e+03 true resid norm 2.114162562951e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.690981568660e-08 true resid norm 1.826808079437e-07 ||r(i)||/||b|| 8.640811787375e-10
>    48 SNES Function norm 2.114160137794e+02 
>       0 KSP preconditioned resid norm 1.136099973228e+04 true resid norm 2.114160137794e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.015775310105e-08 true resid norm 1.261980904678e-07 ||r(i)||/||b|| 5.969183138583e-10
>    49 SNES Function norm 2.114158331092e+02 
>       0 KSP preconditioned resid norm 1.315560026827e+04 true resid norm 2.114158331092e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.131165057283e-08 true resid norm 3.097361970470e-07 ||r(i)||/||b|| 1.465056767470e-09
>    50 SNES Function norm 2.114158223286e+02 
>       0 KSP preconditioned resid norm 1.794717411653e+04 true resid norm 2.114158223286e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.070352092082e-07 true resid norm 6.224398071454e-07 ||r(i)||/||b|| 2.944149592446e-09
>    51 SNES Function norm 2.114157500757e+02 
>       0 KSP preconditioned resid norm 2.077631649185e+04 true resid norm 2.114157500757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.358541701826e-08 true resid norm 5.604873460299e-07 ||r(i)||/||b|| 2.651114431301e-09
>    52 SNES Function norm 2.114156961871e+02 
>       0 KSP preconditioned resid norm 2.405048247398e+04 true resid norm 2.114156961871e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.439555573169e-08 true resid norm 3.896791261159e-07 ||r(i)||/||b|| 1.843189191455e-09
>    53 SNES Function norm 2.114156592651e+02 
>       0 KSP preconditioned resid norm 2.914886710376e+04 true resid norm 2.114156592651e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.436687722935e-07 true resid norm 1.370484810325e-06 ||r(i)||/||b|| 6.482418639609e-09
>    54 SNES Function norm 2.114156319126e+02 
>       0 KSP preconditioned resid norm 3.373835363786e+04 true resid norm 2.114156319126e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.028533154395e-07 true resid norm 6.668879459515e-07 ||r(i)||/||b|| 3.154392794508e-09
>    55 SNES Function norm 2.114156115019e+02 
>       0 KSP preconditioned resid norm 3.904963797200e+04 true resid norm 2.114156115019e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.986152192528e-08 true resid norm 4.508900491313e-07 ||r(i)||/||b|| 2.132718799374e-09
>    56 SNES Function norm 2.114155964342e+02 
>       0 KSP preconditioned resid norm 4.594596937760e+04 true resid norm 2.114155964342e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.567731897537e-08 true resid norm 5.054454305445e-07 ||r(i)||/||b|| 2.390766996709e-09
>    57 SNES Function norm 2.114155854345e+02 
>       0 KSP preconditioned resid norm 5.317614782539e+04 true resid norm 2.114155854345e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.460625544912e-07 true resid norm 9.171801951021e-07 ||r(i)||/||b|| 4.338280894556e-09
>    58 SNES Function norm 2.114155777135e+02 
>       0 KSP preconditioned resid norm 6.400017583847e+04 true resid norm 2.114155777135e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.093685259962e-08 true resid norm 6.506435082837e-07 ||r(i)||/||b|| 3.077557081273e-09
>    59 SNES Function norm 2.114155720468e+02 
>       0 KSP preconditioned resid norm 7.406807436961e+04 true resid norm 2.114155720468e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.571128588743e-07 true resid norm 1.516640238741e-06 ||r(i)||/||b|| 7.173739493538e-09
>    60 SNES Function norm 2.114155680388e+02 
>       0 KSP preconditioned resid norm 8.892466432276e+04 true resid norm 2.114155680388e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.335222631846e-08 true resid norm 8.161361513986e-07 ||r(i)||/||b|| 3.860340839464e-09
>    61 SNES Function norm 2.114155651044e+02 
>       0 KSP preconditioned resid norm 1.029106627628e+05 true resid norm 2.114155651044e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.244108037270e-07 true resid norm 3.009616926997e-06 ||r(i)||/||b|| 1.423555037450e-08
>    62 SNES Function norm 2.114155629136e+02 
>       0 KSP preconditioned resid norm 1.190953464375e+05 true resid norm 2.114155629136e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.108162336627e-08 true resid norm 9.193387836453e-07 ||r(i)||/||b|| 4.348491525295e-09
>    63 SNES Function norm 2.114155612779e+02 
>       0 KSP preconditioned resid norm 1.378243866677e+05 true resid norm 2.114155612779e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.138603793608e-07 true resid norm 2.482322226118e-06 ||r(i)||/||b|| 1.174143573497e-08
>    64 SNES Function norm 2.114155602404e+02 
>       0 KSP preconditioned resid norm 1.698559088929e+05 true resid norm 2.114155602404e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.608494949489e-08 true resid norm 1.125534374356e-06 ||r(i)||/||b|| 5.323801015765e-09
>    65 SNES Function norm 2.114155594364e+02 
>       0 KSP preconditioned resid norm 1.965648734846e+05 true resid norm 2.114155594364e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.689853079655e-07 true resid norm 2.539748843840e-06 ||r(i)||/||b|| 1.201306493529e-08
>    66 SNES Function norm 2.114155588361e+02 
>       0 KSP preconditioned resid norm 2.274726625349e+05 true resid norm 2.114155588361e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.937782777025e-07 true resid norm 5.126870538363e-06 ||r(i)||/||b|| 2.425020450996e-08
>    67 SNES Function norm 2.114155583879e+02 
>       0 KSP preconditioned resid norm 2.632394448333e+05 true resid norm 2.114155583879e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.229084794159e-07 true resid norm 3.026195587359e-06 ||r(i)||/||b|| 1.431396823599e-08
>    68 SNES Function norm 2.114155581765e+02 
>       0 KSP preconditioned resid norm 3.367643572924e+05 true resid norm 2.114155581765e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.632794900620e-08 true resid norm 2.446819364813e-06 ||r(i)||/||b|| 1.157350663271e-08
>    69 SNES Function norm 2.114155579720e+02 
>       0 KSP preconditioned resid norm 3.897127303190e+05 true resid norm 2.114155579720e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.933486330809e-06 true resid norm 2.730183810468e-05 ||r(i)||/||b|| 1.291382638372e-07
>    70 SNES Function norm 2.114155578193e+02 
>       0 KSP preconditioned resid norm 4.509848215072e+05 true resid norm 2.114155578193e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.706389710731e-08 true resid norm 3.785217910694e-06 ||r(i)||/||b|| 1.790415970205e-08
>    71 SNES Function norm 2.114155577054e+02 
>       0 KSP preconditioned resid norm 5.242625554614e+05 true resid norm 2.114155577054e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.321749665453e-06 true resid norm 9.069981074375e-06 ||r(i)||/||b|| 4.290119976419e-08
>    72 SNES Function norm 2.114155576210e+02 
>       0 KSP preconditioned resid norm 6.066871925199e+05 true resid norm 2.114155576210e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.327204488687e-07 true resid norm 7.090997425424e-06 ||r(i)||/||b|| 3.354056581842e-08
>    73 SNES Function norm 2.114155575580e+02 
>       0 KSP preconditioned resid norm 7.020695777561e+05 true resid norm 2.114155575580e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.123816847857e-06 true resid norm 8.125899637337e-06 ||r(i)||/||b|| 3.843567489166e-08
>    74 SNES Function norm 2.114155575109e+02 
>       0 KSP preconditioned resid norm 8.124474305522e+05 true resid norm 2.114155575109e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.293550435673e-07 true resid norm 7.844829319667e-06 ||r(i)||/||b|| 3.710620643072e-08
>    75 SNES Function norm 2.114155574865e+02 
>       0 KSP preconditioned resid norm 1.029314742970e+06 true resid norm 2.114155574865e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.033392384072e-07 true resid norm 9.399925240813e-06 ||r(i)||/||b|| 4.446184260312e-08
>    76 SNES Function norm 2.114155574646e+02 
>       0 KSP preconditioned resid norm 1.191137352032e+06 true resid norm 2.114155574646e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.636526184597e-07 true resid norm 1.077765913224e-05 ||r(i)||/||b|| 5.097855267366e-08
>    77 SNES Function norm 2.114155574575e+02 
>       0 KSP preconditioned resid norm 1.563965951801e+06 true resid norm 2.114155574575e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.137216998536e-06 true resid norm 5.108848518118e-05 ||r(i)||/||b|| 2.416496013612e-07
>    78 SNES Function norm 2.114155574558e+02 
>       0 KSP preconditioned resid norm 2.108895892585e+06 true resid norm 2.114155574558e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.138935971807e-05 true resid norm 6.467637231732e-05 ||r(i)||/||b|| 3.059205911601e-07
>    79 SNES Function norm 2.114155574511e+02 
>       0 KSP preconditioned resid norm 2.564542138115e+06 true resid norm 2.114155574511e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.391129540980e-06 true resid norm 4.283623913226e-05 ||r(i)||/||b|| 2.026163052933e-07
>    80 SNES Function norm 2.114155574475e+02 
>       0 KSP preconditioned resid norm 2.967732587697e+06 true resid norm 2.114155574475e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.446508619620e-06 true resid norm 5.164837794317e-05 ||r(i)||/||b|| 2.442979058246e-07
>    81 SNES Function norm 2.114155574449e+02 
>       0 KSP preconditioned resid norm 3.434290246612e+06 true resid norm 2.114155574449e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.781979196562e-06 true resid norm 4.886329177979e-05 ||r(i)||/||b|| 2.311243901364e-07
>    82 SNES Function norm 2.114155574429e+02 
>       0 KSP preconditioned resid norm 3.974205574181e+06 true resid norm 2.114155574429e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.425678262963e-05 true resid norm 8.517034218173e-05 ||r(i)||/||b|| 4.028574964485e-07
>    83 SNES Function norm 2.114155574416e+02 
>       0 KSP preconditioned resid norm 4.785116089506e+06 true resid norm 2.114155574416e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.130667094145e-05 true resid norm 1.264608296603e-04 ||r(i)||/||b|| 5.981623641639e-07
>    84 SNES Function norm 2.114155574406e+02 
>       0 KSP preconditioned resid norm 5.537350742612e+06 true resid norm 2.114155574406e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.299149833543e-06 true resid norm 5.419308321331e-05 ||r(i)||/||b|| 2.563344148812e-07
>    85 SNES Function norm 2.114155574399e+02 
>       0 KSP preconditioned resid norm 6.767161611151e+06 true resid norm 2.114155574399e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.781986133631e-05 true resid norm 1.232616969009e-04 ||r(i)||/||b|| 5.830303994346e-07
>    86 SNES Function norm 2.114155574394e+02 
>       0 KSP preconditioned resid norm 7.867884017146e+06 true resid norm 2.114155574394e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.002696664504e-05 true resid norm 1.398533205346e-04 ||r(i)||/||b|| 6.615091255747e-07
>    87 SNES Function norm 2.114155574390e+02 
>       0 KSP preconditioned resid norm 9.102289060164e+06 true resid norm 2.114155574390e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.231201322143e-05 true resid norm 1.368407286352e-04 ||r(i)||/||b|| 6.472595030036e-07
>    88 SNES Function norm 2.114155574387e+02 
>       0 KSP preconditioned resid norm 1.063720954841e+07 true resid norm 2.114155574387e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.517853677131e-05 true resid norm 1.273061147947e-04 ||r(i)||/||b|| 6.021605805032e-07
>    89 SNES Function norm 2.114155574385e+02 
>       0 KSP preconditioned resid norm 1.230866787650e+07 true resid norm 2.114155574385e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.745229646688e-05 true resid norm 1.389443348166e-04 ||r(i)||/||b|| 6.572096041560e-07
>    90 SNES Function norm 2.114155574384e+02 
>       0 KSP preconditioned resid norm 1.436772950622e+07 true resid norm 2.114155574384e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.955301916175e-05 true resid norm 5.633754040143e-04 ||r(i)||/||b|| 2.664777421494e-06
>    91 SNES Function norm 2.114155574383e+02 
>       0 KSP preconditioned resid norm 1.662010558966e+07 true resid norm 2.114155574383e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.147116976734e-05 true resid norm 1.351152564949e-04 ||r(i)||/||b|| 6.390979837628e-07
>    92 SNES Function norm 2.114155574382e+02 
>       0 KSP preconditioned resid norm 1.923255924422e+07 true resid norm 2.114155574382e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.030030646485e-05 true resid norm 1.877965090103e-04 ||r(i)||/||b|| 8.882814078868e-07
>    93 SNES Function norm 2.114155574381e+02 
>       0 KSP preconditioned resid norm 2.226793244995e+07 true resid norm 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.411985876631e-04 true resid norm 1.342893408929e-03 ||r(i)||/||b|| 6.351913857247e-06
>    94 SNES Function norm 2.114155574381e+02 
>       0 KSP preconditioned resid norm 2.576953439486e+07 true resid norm 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.147595096922e-05 true resid norm 4.578229507611e-04 ||r(i)||/||b|| 2.165512114193e-06
>    95 SNES Function norm 2.114155574380e+02 
>       0 KSP preconditioned resid norm 2.981662207354e+07 true resid norm 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.456246888386e-06 true resid norm 2.856742906943e-04 ||r(i)||/||b|| 1.351245358460e-06
>    96 SNES Function norm 2.114155574380e+02 
>       0 KSP preconditioned resid norm 3.451395888978e+07 true resid norm 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.972976681165e-05 true resid norm 3.126288732572e-04 ||r(i)||/||b|| 1.478741096662e-06
>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 96
> 
> -gideon
> 
>> On Sep 8, 2015, at 10:31 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Tue, Sep 8, 2015 at 9:19 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> Ok, going back and running the problem with the flags, -snes_fd -snes_mf_operator -ksp_monitor_true_residual -snes_max_funcs 500000, during the grid sequence, I still get:
>> 
>> Lets try using -pc_type lu instead of MUMPS, just to make sure. If we still
>> get this, then the problem is really horribly conditioned without being singular.
>> We are losing all precision for some reason. Is it possible that the two halves
>> are so differently scaled that we make no progress on one with LU?
>> 
>>   Thanks,
>> 
>>      Matt
>>  
>>       0 SNES Function norm 9.131726454160e-09 
>>         0 KSP preconditioned resid norm 4.201012306235e-09 true resid norm 9.131726454160e-09 ||r(i)||/||b|| 1.000000000000e+00
>>         1 KSP preconditioned resid norm 4.811278613811e-14 true resid norm 7.117495013042e-13 ||r(i)||/||b|| 7.794249037978e-05
>>         2 KSP preconditioned resid norm 1.729322442321e-19 true resid norm 2.611605096203e-13 ||r(i)||/||b|| 2.859924800982e-05
>>       1 SNES Function norm 2.066497600915e-12 
>>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>     0 SNES Function norm 5.066222213176e+03 
>>       0 KSP preconditioned resid norm 3.135051257976e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.425321207844e-04 true resid norm 1.654961474093e+00 ||r(i)||/||b|| 3.266657885218e-04
>>       2 KSP preconditioned resid norm 2.739130995469e-09 true resid norm 7.943827483388e-03 ||r(i)||/||b|| 1.567998234015e-06
>>     1 SNES Function norm 8.487402654330e+02 
>>       0 KSP preconditioned resid norm 2.670376525819e+02 true resid norm 8.487402654330e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.518628784729e-02 true resid norm 1.820466086409e+00 ||r(i)||/||b|| 2.144903641964e-03
>>       2 KSP preconditioned resid norm 1.680654269556e-07 true resid norm 1.635299027240e-01 ||r(i)||/||b|| 1.926736710678e-04
>>     2 SNES Function norm 6.547660268557e+02 
>>       0 KSP preconditioned resid norm 1.529984375428e+02 true resid norm 6.547660268557e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.106652647444e-02 true resid norm 9.350438798847e-01 ||r(i)||/||b|| 1.428058026124e-03
>>       2 KSP preconditioned resid norm 1.739404692247e-07 true resid norm 6.642022841611e-02 ||r(i)||/||b|| 1.014411647701e-04
>>     3 SNES Function norm 5.771035131885e+02 
>>       0 KSP preconditioned resid norm 1.360697738776e+02 true resid norm 5.771035131885e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.323138791370e-03 true resid norm 6.419077863137e-02 ||r(i)||/||b|| 1.112292286642e-04
>>     4 SNES Function norm 5.238679723156e+02 
>>       0 KSP preconditioned resid norm 1.528156345475e+02 true resid norm 5.238679723156e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.115694692015e-02 true resid norm 1.230231531838e+00 ||r(i)||/||b|| 2.348361795054e-03
>>       2 KSP preconditioned resid norm 5.444815701049e-08 true resid norm 6.640979326953e-02 ||r(i)||/||b|| 1.267681873660e-04
>>     5 SNES Function norm 4.755828064476e+02 
>>       0 KSP preconditioned resid norm 1.872910183528e+02 true resid norm 4.755828064476e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.182180755863e-02 true resid norm 5.918459188490e-01 ||r(i)||/||b|| 1.244464498769e-03
>>       2 KSP preconditioned resid norm 7.426717097996e-08 true resid norm 8.473303784961e-02 ||r(i)||/||b|| 1.781667392111e-04
>>     6 SNES Function norm 4.224257767287e+02 
>>       0 KSP preconditioned resid norm 2.578229460575e+02 true resid norm 4.224257767287e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.258348180527e-02 true resid norm 3.257118054248e-01 ||r(i)||/||b|| 7.710509712432e-04
>>       2 KSP preconditioned resid norm 1.122972268815e-07 true resid norm 1.274729638407e-01 ||r(i)||/||b|| 3.017641698568e-04
>>     7 SNES Function norm 3.811223987907e+02 
>>       0 KSP preconditioned resid norm 5.218684581634e+02 true resid norm 3.811223987907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.089005298384e-01 true resid norm 5.194667956652e+00 ||r(i)||/||b|| 1.362992039601e-02
>>       2 KSP preconditioned resid norm 7.570453480881e-07 true resid norm 2.863356359719e-01 ||r(i)||/||b|| 7.512957435207e-04
>>     8 SNES Function norm 3.771290745624e+02 
>>       0 KSP preconditioned resid norm 5.827455360446e+03 true resid norm 3.771290745624e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.620225652579e-01 true resid norm 1.332007773441e+01 ||r(i)||/||b|| 3.531967867995e-02
>>       2 KSP preconditioned resid norm 3.596801636560e-06 true resid norm 3.425465452374e+00 ||r(i)||/||b|| 9.083005483862e-03
>>     9 SNES Function norm 3.767897776540e+02 
>>       0 KSP preconditioned resid norm 5.185115189077e+03 true resid norm 3.767897776540e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.021105455600e-01 true resid norm 1.043813038739e+01 ||r(i)||/||b|| 2.770279611187e-02
>>       2 KSP preconditioned resid norm 6.951879108628e-06 true resid norm 2.816562934830e+00 ||r(i)||/||b|| 7.475157506572e-03
>>    10 SNES Function norm 3.746968158051e+02 
>>       0 KSP preconditioned resid norm 8.165048723911e+02 true resid norm 3.746968158051e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.177528145230e-01 true resid norm 2.394044923481e+00 ||r(i)||/||b|| 6.389285476945e-03
>>       2 KSP preconditioned resid norm 7.325770200602e-07 true resid norm 4.877299135281e-01 ||r(i)||/||b|| 1.301665487816e-03
>>    11 SNES Function norm 3.726877791108e+02 
>>       0 KSP preconditioned resid norm 5.024547551936e+02 true resid norm 3.726877791108e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.640017871568e-01 true resid norm 1.648101091752e+00 ||r(i)||/||b|| 4.422203206351e-03
>>       2 KSP preconditioned resid norm 1.005738633834e-06 true resid norm 2.935508517682e-01 ||r(i)||/||b|| 7.876589151074e-04
>>    12 SNES Function norm 3.668832918639e+02 
>>       0 KSP preconditioned resid norm 7.762104953873e+02 true resid norm 3.668832918639e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.912259329681e-02 true resid norm 1.334600681831e+00 ||r(i)||/||b|| 3.637670919956e-03
>>       2 KSP preconditioned resid norm 7.640320635302e-07 true resid norm 4.471334143387e-01 ||r(i)||/||b|| 1.218734742776e-03
>>    13 SNES Function norm 3.631997889667e+02 
>>       0 KSP preconditioned resid norm 2.515761606628e+02 true resid norm 3.631997889667e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.648297544435e-01 true resid norm 2.866637955219e+00 ||r(i)||/||b|| 7.892730233611e-03
>>       2 KSP preconditioned resid norm 7.305731711999e-07 true resid norm 1.472589913546e-01 ||r(i)||/||b|| 4.054490003243e-04
>>    14 SNES Function norm 3.557844076846e+02 
>>       0 KSP preconditioned resid norm 6.995145168850e+02 true resid norm 3.557844076846e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.626527197886e-02 true resid norm 5.642316830909e-01 ||r(i)||/||b|| 1.585880861848e-03
>>       2 KSP preconditioned resid norm 1.040702263139e-06 true resid norm 4.073703976052e-01 ||r(i)||/||b|| 1.144992272866e-03
>>    15 SNES Function norm 3.534147071188e+02 
>>       0 KSP preconditioned resid norm 2.698507093385e+02 true resid norm 3.534147071188e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.592052963868e-01 true resid norm 1.332346399928e+00 ||r(i)||/||b|| 3.769923472595e-03
>>       2 KSP preconditioned resid norm 5.021635239169e-07 true resid norm 1.573071613558e-01 ||r(i)||/||b|| 4.451064378114e-04
>>    16 SNES Function norm 3.499900270010e+02 
>>       0 KSP preconditioned resid norm 7.295084253442e+01 true resid norm 3.499900270010e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.040248007780e-02 true resid norm 1.427506717563e-01 ||r(i)||/||b|| 4.078706841436e-04
>>       2 KSP preconditioned resid norm 1.246491224109e-07 true resid norm 4.074099442269e-02 ||r(i)||/||b|| 1.164061581177e-04
>>    17 SNES Function norm 3.473017914657e+02 
>>       0 KSP preconditioned resid norm 4.169372990718e+02 true resid norm 3.473017914657e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.140552389441e-01 true resid norm 1.893314948416e+00 ||r(i)||/||b|| 5.451497789361e-03
>>       2 KSP preconditioned resid norm 7.751532205539e-07 true resid norm 2.494992549211e-01 ||r(i)||/||b|| 7.183932275966e-04
>>    18 SNES Function norm 3.464416771335e+02 
>>       0 KSP preconditioned resid norm 2.968698304201e+02 true resid norm 3.464416771335e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.683113115222e-02 true resid norm 2.501212385243e-01 ||r(i)||/||b|| 7.219721385540e-04
>>       2 KSP preconditioned resid norm 5.274081741342e-07 true resid norm 1.712806350336e-01 ||r(i)||/||b|| 4.943996243489e-04
>>    19 SNES Function norm 3.460747441983e+02 
>>       0 KSP preconditioned resid norm 3.707198448214e+02 true resid norm 3.460747441983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.740664131652e-02 true resid norm 9.477346570578e-01 ||r(i)||/||b|| 2.738525919461e-03
>>       2 KSP preconditioned resid norm 4.649501179917e-07 true resid norm 2.217443031583e-01 ||r(i)||/||b|| 6.407410736429e-04
>>    20 SNES Function norm 3.455596398964e+02 
>>       0 KSP preconditioned resid norm 5.000331867417e+02 true resid norm 3.455596398964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.821124737573e-02 true resid norm 7.651194558377e-01 ||r(i)||/||b|| 2.214145888296e-03
>>       2 KSP preconditioned resid norm 5.030283955029e-07 true resid norm 2.918161448112e-01 ||r(i)||/||b|| 8.444740389784e-04
>>    21 SNES Function norm 3.453222949611e+02 
>>       0 KSP preconditioned resid norm 7.399233760260e+02 true resid norm 3.453222949611e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.030913689264e-01 true resid norm 8.930798988128e-01 ||r(i)||/||b|| 2.586221370136e-03
>>       2 KSP preconditioned resid norm 1.497261432173e-06 true resid norm 4.385255355661e-01 ||r(i)||/||b|| 1.269902181136e-03
>>    22 SNES Function norm 3.452850731057e+02 
>>       0 KSP preconditioned resid norm 1.850020325701e+03 true resid norm 3.452850731057e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.670597549481e-01 true resid norm 6.205758169419e+00 ||r(i)||/||b|| 1.797285389027e-02
>>       2 KSP preconditioned resid norm 4.122440586032e-06 true resid norm 1.109676174925e+00 ||r(i)||/||b|| 3.213797124053e-03
>>    23 SNES Function norm 3.452722963704e+02 
>>       0 KSP preconditioned resid norm 3.362446711623e+03 true resid norm 3.452722963704e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.181412468129e+00 true resid norm 1.564006484993e+01 ||r(i)||/||b|| 4.529776936738e-02
>>       2 KSP preconditioned resid norm 1.165193864469e-05 true resid norm 1.869411988347e+00 ||r(i)||/||b|| 5.414312147250e-03
>>    24 SNES Function norm 3.452679622612e+02 
>>       0 KSP preconditioned resid norm 6.149140486077e+03 true resid norm 3.452679622612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.400761945257e-01 true resid norm 3.795886928810e+00 ||r(i)||/||b|| 1.099403171945e-02
>>       2 KSP preconditioned resid norm 1.754461070240e-06 true resid norm 3.648059240080e+00 ||r(i)||/||b|| 1.056587821294e-02
>>    25 SNES Function norm 3.452667776779e+02 
>>       0 KSP preconditioned resid norm 1.101683177789e+04 true resid norm 3.452667776779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.317993935110e+00 true resid norm 2.454711939414e+01 ||r(i)||/||b|| 7.109609432809e-02
>>       2 KSP preconditioned resid norm 3.457477291813e-05 true resid norm 6.807469339003e+00 ||r(i)||/||b|| 1.971654899665e-02
>>    26 SNES Function norm 3.452666586474e+02 
>>       0 KSP preconditioned resid norm 3.879190850784e+04 true resid norm 3.452666586474e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.739562350939e+01 true resid norm 1.221587956319e+02 ||r(i)||/||b|| 3.538099974971e-01
>>       2 KSP preconditioned resid norm 7.455463420492e-05 true resid norm 3.080916578781e+01 ||r(i)||/||b|| 8.923295955800e-02
>>    27 SNES Function norm 3.452666425264e+02 
>>       0 KSP preconditioned resid norm 5.709318799171e+04 true resid norm 3.452666425264e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.292899448408e-01 true resid norm 2.801354737138e+01 ||r(i)||/||b|| 8.113597990933e-02
>>    28 SNES Function norm 3.452666374187e+02 
>>       0 KSP preconditioned resid norm 2.488913348294e+05 true resid norm 3.452666374187e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.691998534170e+02 true resid norm 7.244072115265e+03 ||r(i)||/||b|| 2.098109498625e+01
>>       2 KSP preconditioned resid norm 2.503929513070e-03 true resid norm 2.928791518636e+03 ||r(i)||/||b|| 8.482694825461e+00
>>    29 SNES Function norm 3.452666374177e+02 
>>       0 KSP preconditioned resid norm 2.533150641403e+05 true resid norm 3.452666374177e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.817060595281e+02 true resid norm 1.333479168158e+03 ||r(i)||/||b|| 3.862172082803e+00
>>       2 KSP preconditioned resid norm 8.930133492283e-04 true resid norm 5.412764197416e+02 ||r(i)||/||b|| 1.567705538507e+00
>>    30 SNES Function norm 3.452666373128e+02 
>>       0 KSP preconditioned resid norm 2.169409532487e+05 true resid norm 3.452666373128e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.325438416139e+02 true resid norm 9.441618139691e+02 ||r(i)||/||b|| 2.734587451940e+00
>>       2 KSP preconditioned resid norm 8.796591247860e-04 true resid norm 2.978002325996e+02 ||r(i)||/||b|| 8.625224693509e-01
>>    31 SNES Function norm 3.452666372449e+02 
>>       0 KSP preconditioned resid norm 1.459475274582e+05 true resid norm 3.452666372449e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.978987062939e+02 true resid norm 1.411576222768e+03 ||r(i)||/||b|| 4.088365542735e+00
>>       2 KSP preconditioned resid norm 1.235854958341e-03 true resid norm 3.579390387115e+02 ||r(i)||/||b|| 1.036703231936e+00
>>    32 SNES Function norm 3.452666369928e+02 
>>       0 KSP preconditioned resid norm 1.264234515603e+05 true resid norm 3.452666369928e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.676975931928e+01 true resid norm 1.855746501202e+02 ||r(i)||/||b|| 5.374821376792e-01
>>       2 KSP preconditioned resid norm 9.286140031498e-05 true resid norm 3.495715551553e+01 ||r(i)||/||b|| 1.012468387331e-01
>>    33 SNES Function norm 3.452666245901e+02 
>>       0 KSP preconditioned resid norm 1.544591438108e+06 true resid norm 3.452666245901e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.501087626485e+01 true resid norm 3.233291920369e+02 ||r(i)||/||b|| 9.364623424597e-01
>>       2 KSP preconditioned resid norm 1.867215855867e-04 true resid norm 7.114686634655e+01 ||r(i)||/||b|| 2.060635499623e-01
>>    34 SNES Function norm 3.452666216061e+02 
>>       0 KSP preconditioned resid norm 3.431570281248e+05 true resid norm 3.452666216061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.140176766342e+02 true resid norm 1.760366951993e+03 ||r(i)||/||b|| 5.098572644538e+00
>>       2 KSP preconditioned resid norm 2.324744486242e-03 true resid norm 1.080605579953e+03 ||r(i)||/||b|| 3.129771348664e+00
>>    35 SNES Function norm 3.452666215921e+02 
>>       0 KSP preconditioned resid norm 3.734063885343e+05 true resid norm 3.452666215921e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.279219133674e+00 true resid norm 1.691481367261e+02 ||r(i)||/||b|| 4.899058470990e-01
>>       2 KSP preconditioned resid norm 4.046466703016e-04 true resid norm 1.964786369469e+02 ||r(i)||/||b|| 5.690635139908e-01
>>    36 SNES Function norm 3.452666212200e+02 
>>       0 KSP preconditioned resid norm 7.952390829283e+05 true resid norm 3.452666212200e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.707607129733e+01 true resid norm 2.806640653978e+02 ||r(i)||/||b|| 8.128908158167e-01
>>       2 KSP preconditioned resid norm 1.296391885225e-04 true resid norm 1.079623998255e+02 ||r(i)||/||b|| 3.126928384910e-01
>>    37 SNES Function norm 3.452666199636e+02 
>>       0 KSP preconditioned resid norm 9.358626316710e+05 true resid norm 3.452666199636e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.722696571742e+01 true resid norm 3.251492717238e+02 ||r(i)||/||b|| 9.417338744130e-01
>>       2 KSP preconditioned resid norm 2.862108374145e-04 true resid norm 3.320104402606e+01 ||r(i)||/||b|| 9.616059620694e-02
>>    38 SNES Function norm 3.452666133867e+02 
>>       0 KSP preconditioned resid norm 8.365871848022e+04 true resid norm 3.452666133867e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.482543031390e+02 true resid norm 1.772891732407e+03 ||r(i)||/||b|| 5.134848443690e+00
>>       2 KSP preconditioned resid norm 1.439398367387e-03 true resid norm 3.088902452162e+02 ||r(i)||/||b|| 8.946426710257e-01
>>    39 SNES Function norm 3.452666132332e+02 
>>       0 KSP preconditioned resid norm 8.960112658041e+04 true resid norm 3.452666132332e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.505891399150e+01 true resid norm 1.787713293837e+02 ||r(i)||/||b|| 5.177776319280e-01
>>       2 KSP preconditioned resid norm 6.384856870578e-05 true resid norm 2.543851208517e+01 ||r(i)||/||b|| 7.367787996341e-02
>>    40 SNES Function norm 3.452665904125e+02 
>>       0 KSP preconditioned resid norm 7.796808144307e+05 true resid norm 3.452665904125e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.020764189760e+01 true resid norm 6.961545395677e+02 ||r(i)||/||b|| 2.016281212544e+00
>>       2 KSP preconditioned resid norm 3.506849883654e-04 true resid norm 4.956460306859e+02 ||r(i)||/||b|| 1.435545878024e+00
>>    41 SNES Function norm 3.452665902941e+02 
>>       0 KSP preconditioned resid norm 4.990797566798e+05 true resid norm 3.452665902941e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.012324354359e+02 true resid norm 7.390530765626e+02 ||r(i)||/||b|| 2.140528789458e+00
>>       2 KSP preconditioned resid norm 7.111862680292e-04 true resid norm 4.066260265930e+02 ||r(i)||/||b|| 1.177716112777e+00
>>    42 SNES Function norm 3.452665901159e+02 
>>       0 KSP preconditioned resid norm 3.472654025442e+05 true resid norm 3.452665901159e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.552680791160e+02 true resid norm 1.064105325830e+03 ||r(i)||/||b|| 3.081981739016e+00
>>       2 KSP preconditioned resid norm 2.695346315896e-03 true resid norm 5.359619569435e+02 ||r(i)||/||b|| 1.552313407340e+00
>>    43 SNES Function norm 3.452665900083e+02 
>>       0 KSP preconditioned resid norm 2.823873507449e+05 true resid norm 3.452665900083e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.616335635707e+00 true resid norm 1.564312339233e+02 ||r(i)||/||b|| 4.530737651723e-01
>>       2 KSP preconditioned resid norm 1.580836053583e-04 true resid norm 1.777870114826e+02 ||r(i)||/||b|| 5.149267743465e-01
>>    44 SNES Function norm 3.452665895210e+02 
>>       0 KSP preconditioned resid norm 4.764215602672e+05 true resid norm 3.452665895210e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.491949190147e+01 true resid norm 4.652947485976e+02 ||r(i)||/||b|| 1.347639078670e+00
>>       2 KSP preconditioned resid norm 2.266106888017e-04 true resid norm 1.316722015429e+02 ||r(i)||/||b|| 3.813638664707e-01
>>    45 SNES Function norm 3.452665889075e+02 
>>       0 KSP preconditioned resid norm 1.439659890301e+05 true resid norm 3.452665889075e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.431461947162e+01 true resid norm 1.403614551576e+02 ||r(i)||/||b|| 4.065306625865e-01
>>       2 KSP preconditioned resid norm 4.711117728059e-04 true resid norm 1.095117712535e+02 ||r(i)||/||b|| 3.171803318704e-01
>>    46 SNES Function norm 3.452665876547e+02 
>>       0 KSP preconditioned resid norm 2.157313334766e+05 true resid norm 3.452665876547e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.967750174401e+00 true resid norm 1.053591658730e+02 ||r(i)||/||b|| 3.051530893525e-01
>>       2 KSP preconditioned resid norm 4.512059841816e-04 true resid norm 1.189061020519e+02 ||r(i)||/||b|| 3.443892525473e-01
>>    47 SNES Function norm 3.452665866229e+02 
>>       0 KSP preconditioned resid norm 3.981118078251e+05 true resid norm 3.452665866229e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.013119377232e+01 true resid norm 4.969789311062e+02 ||r(i)||/||b|| 1.439406390196e+00
>>       2 KSP preconditioned resid norm 1.872910444347e-04 true resid norm 1.396396719776e+02 ||r(i)||/||b|| 4.044401554852e-01
>>    48 SNES Function norm 3.452665850954e+02 
>>       0 KSP preconditioned resid norm 1.967554630062e+05 true resid norm 3.452665850954e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.006970327024e+00 true resid norm 9.616098906643e+01 ||r(i)||/||b|| 2.785122951874e-01
>>       2 KSP preconditioned resid norm 3.184368801558e-05 true resid norm 9.951979756268e+01 ||r(i)||/||b|| 2.882404549377e-01
>>    49 SNES Function norm 3.452665835600e+02 
>>       0 KSP preconditioned resid norm 3.965648247310e+05 true resid norm 3.452665835600e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.862969730098e+01 true resid norm 1.790616593203e+02 ||r(i)||/||b|| 5.186185627174e-01
>>       2 KSP preconditioned resid norm 2.964457441143e-04 true resid norm 1.447818294544e+02 ||r(i)||/||b|| 4.193334552146e-01
>>    50 SNES Function norm 3.452665828612e+02 
>>       0 KSP preconditioned resid norm 1.265022286683e+06 true resid norm 3.452665828612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.401066920177e+01 true resid norm 3.143671803790e+02 ||r(i)||/||b|| 9.105056671685e-01
>>       2 KSP preconditioned resid norm 1.444898193175e-04 true resid norm 7.578180059616e+01 ||r(i)||/||b|| 2.194877939480e-01
>>    51 SNES Function norm 3.452665817118e+02 
>>       0 KSP preconditioned resid norm 2.063334812849e+05 true resid norm 3.452665817118e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.074628697499e+01 true resid norm 1.155569679685e+02 ||r(i)||/||b|| 3.346891187543e-01
>>       2 KSP preconditioned resid norm 1.326420665890e-04 true resid norm 9.261090091148e+01 ||r(i)||/||b|| 2.682301323584e-01
>>    52 SNES Function norm 3.452665807811e+02 
>>       0 KSP preconditioned resid norm 6.549963238965e+06 true resid norm 3.452665807811e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.246352248726e+01 true resid norm 3.604213542889e+02 ||r(i)||/||b|| 1.043892963731e+00
>>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 52
>> 
>> which is a little different, but not so much different, than what i get if I use my Jacobian.
>> 
>> -gideon
>> 
>>> On Sep 8, 2015, at 10:00 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> Yup, now it?s working.
>>> -gideon
>>> 
>>>> On Sep 8, 2015, at 9:58 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>> 
>>>> On Tue, Sep 8, 2015 at 8:51 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> Going back to this point, a case that previously worked, is now giving me problems that I don?t understand:
>>>> 
>>>>   0 SNES Function norm 1.421454390131e-02 
>>>>     0 KSP preconditioned resid norm 7.635339584553e-03 true resid norm 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.838956718317e-07 true resid norm 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
>>>>     2 KSP preconditioned resid norm 5.889748199467e-12 true resid norm 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
>>>>   1 SNES Function norm 4.476139939604e-06 
>>>>     0 KSP preconditioned resid norm 4.897102387844e-05 true resid norm 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.133649519960e-09 true resid norm 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
>>>>     2 KSP preconditioned resid norm 2.223328042462e-14 true resid norm 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
>>>>   2 SNES Function norm 5.447761863330e-09 
>>>>     0 KSP preconditioned resid norm 9.703890346186e-09 true resid norm 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.982949185218e-13 true resid norm 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
>>>>     2 KSP preconditioned resid norm 7.045835741315e-18 true resid norm 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
>>>> Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT iterations 2
>>>> 
>>>> Somehow the maximum number of function evaluations is being exceeded. Oh, are you using -snes_fd? This uses a function evaluation
>>>> for each column, so you might have to up it > 10000. The convergence is fine.
>>>> 
>>>>   Thanks,
>>>> 
>>>>      Matt
>>>>  
>>>> -gideon
>>>> 
>>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>>>> 
>>>>>  Barry
>>>>> 
>>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>> 
>>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>>>>> 
>>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>>>> 
>>>>>>>   Matt
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>>>> 
>>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>>>> 
>>>>>>>> Gideon,
>>>>>>>> 
>>>>>>>>  In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>>>> 
>>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>>>>> 
>>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>>>> 
>>>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>>>> 
>>>>>>>>>>>   0 SNES Function norm 3.857327250267e-09 
>>>>>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>>>   1 SNES Function norm 2.479697868412e-12 
>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>>>> 
>>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>>>> 
>>>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>  0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>>>>  1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> then 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>>>>    type: seqaij
>>>>>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>>>      MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>>>>        PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>>>>          [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0]  143924 
>>>>>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0] 15991 
>>>>>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>>>    SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>>>    etc
>>>>>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 


From knepley at gmail.com  Tue Sep  8 21:53:56 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Sep 2015 21:53:56 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
Message-ID: <CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>

On Tue, Sep 8, 2015 at 9:48 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Your comment about the different pieces scaling different made me think of
> something, and I made a slight modification.  It still fails, but the
> conditioning looks much better, no?
>

This is great. Now its clear that the nonlinear convergence is crappy.

Lets start with looking at the distribution of residuals. Take iterate 50,
which is as converged as you get and save the residual (VecView with
binary). Then we can load it up and play with it. I recommend

  a) Norms for both halves

  b) Plot of the big half

What usually happens is that some small part is screwing up, but you
have to iterate the whole system, and finding the descent direction is
hard. If you select the problem variables, Newton will converge, and then
the big system will converge once the problem is removed. This is the
classic
"coarse basis selection" problem in the nonlinear regime.

   Matt

  Thanks,

    Matt


>
>       0 SNES Function norm 4.027474333769e-09
>         0 KSP preconditioned resid norm 9.390682304142e-09 true resid norm
> 4.027474333769e-09 ||r(i)||/||b|| 1.000000000000e+00
>         1 KSP preconditioned resid norm 4.937459249116e-20 true resid norm
> 3.737772204700e-19 ||r(i)||/||b|| 9.280685350023e-11
>       1 SNES Function norm 2.666655509899e-12
>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>     0 SNES Function norm 5.066222213175e+03
>       0 KSP preconditioned resid norm 3.135087050015e+01 true resid norm
> 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.220576053756e-11 true resid norm
> 1.206309587478e-07 ||r(i)||/||b|| 2.381083056997e-11
>     1 SNES Function norm 8.484697184819e+02
>       0 KSP preconditioned resid norm 2.668083368240e+02 true resid norm
> 8.484697184819e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.388166746665e-10 true resid norm
> 2.735732864000e-08 ||r(i)||/||b|| 3.224314085004e-11
>     2 SNES Function norm 6.549559723821e+02
>       0 KSP preconditioned resid norm 1.552154273063e+02 true resid norm
> 6.549559723821e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.740227244709e-10 true resid norm
> 1.800722435354e-08 ||r(i)||/||b|| 2.749379364852e-11
>     3 SNES Function norm 5.770723278635e+02
>       0 KSP preconditioned resid norm 1.348596653013e+02 true resid norm
> 5.770723278635e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.142365551700e-10 true resid norm
> 7.387664544933e-09 ||r(i)||/||b|| 1.280197331985e-11
>     4 SNES Function norm 5.237702241066e+02
>       0 KSP preconditioned resid norm 1.524401223470e+02 true resid norm
> 5.237702241066e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.168592748079e-10 true resid norm
> 9.512653260875e-09 ||r(i)||/||b|| 1.816188248788e-11
>     5 SNES Function norm 4.753909020302e+02
>       0 KSP preconditioned resid norm 1.864261890160e+02 true resid norm
> 4.753909020302e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.744912816503e-10 true resid norm
> 1.889384889937e-08 ||r(i)||/||b|| 3.974381675939e-11
>     6 SNES Function norm 4.221784591168e+02
>       0 KSP preconditioned resid norm 2.550992140633e+02 true resid norm
> 4.221784591168e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.697718851690e-10 true resid norm
> 1.666255394677e-08 ||r(i)||/||b|| 3.946803439861e-11
>     7 SNES Function norm 3.806525080909e+02
>       0 KSP preconditioned resid norm 5.183605288372e+02 true resid norm
> 3.806525080909e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.240268561213e-09 true resid norm
> 1.041310587019e-07 ||r(i)||/||b|| 2.735593657958e-10
>     8 SNES Function norm 3.762054656363e+02
>       0 KSP preconditioned resid norm 5.320776249886e+03 true resid norm
> 3.762054656363e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.328856456273e-09 true resid norm
> 7.860313076537e-08 ||r(i)||/||b|| 2.089367059897e-10
>     9 SNES Function norm 3.758975227207e+02
>       0 KSP preconditioned resid norm 6.252711400154e+03 true resid norm
> 3.758975227207e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.900963654627e-09 true resid norm
> 1.347299589292e-07 ||r(i)||/||b|| 3.584220453330e-10
>    10 SNES Function norm 3.757032042008e+02
>       0 KSP preconditioned resid norm 5.890497230519e+02 true resid norm
> 3.757032042008e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.497822807230e-09 true resid norm
> 3.605311757144e-08 ||r(i)||/||b|| 9.596169840535e-11
>    11 SNES Function norm 3.728798164183e+02
>       0 KSP preconditioned resid norm 1.081356468099e+03 true resid norm
> 3.728798164183e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.713971332514e-09 true resid norm
> 3.705072372867e-08 ||r(i)||/||b|| 9.936371478768e-11
>    12 SNES Function norm 3.723078736486e+02
>       0 KSP preconditioned resid norm 3.745123875883e+03 true resid norm
> 3.723078736486e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.885886422519e-08 true resid norm
> 2.837206039485e-07 ||r(i)||/||b|| 7.620591022374e-10
>    13 SNES Function norm 3.721848051981e+02
>       0 KSP preconditioned resid norm 3.495480420016e+03 true resid norm
> 3.721848051981e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.874759607493e-08 true resid norm
> 2.751735241072e-07 ||r(i)||/||b|| 7.393464759014e-10
>    14 SNES Function norm 3.720227571463e+02
>       0 KSP preconditioned resid norm 1.265998097646e+04 true resid norm
> 3.720227571463e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.125038941589e-08 true resid norm
> 5.141611324046e-07 ||r(i)||/||b|| 1.382069033488e-09
>    15 SNES Function norm 3.720051993666e+02
>       0 KSP preconditioned resid norm 2.912369751490e+03 true resid norm
> 3.720051993666e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.170907138699e-09 true resid norm
> 3.623872061163e-08 ||r(i)||/||b|| 9.741455407971e-11
>    16 SNES Function norm 3.718945615434e+02
>       0 KSP preconditioned resid norm 1.082053028195e+03 true resid norm
> 3.718945615434e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.660982704773e-09 true resid norm
> 9.903133228018e-08 ||r(i)||/||b|| 2.662887348210e-10
>    17 SNES Function norm 3.700412140239e+02
>       0 KSP preconditioned resid norm 3.676306553630e+01 true resid norm
> 3.700412140239e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.389666775474e-10 true resid norm
> 4.384613797840e-10 ||r(i)||/||b|| 1.184898771183e-12
>    18 SNES Function norm 3.351961239413e+02
>       0 KSP preconditioned resid norm 1.071519127966e+02 true resid norm
> 3.351961239413e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.391049839274e-11 true resid norm
> 1.771280721778e-09 ||r(i)||/||b|| 5.284311468017e-12
>    19 SNES Function norm 3.096040290055e+02
>       0 KSP preconditioned resid norm 3.362840348158e+02 true resid norm
> 3.096040290055e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.847034649159e-10 true resid norm
> 3.697317355249e-09 ||r(i)||/||b|| 1.194208411023e-11
>    20 SNES Function norm 3.008687462984e+02
>       0 KSP preconditioned resid norm 7.161404453567e+01 true resid norm
> 3.008687462984e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.352839847013e-11 true resid norm
> 7.837384020140e-10 ||r(i)||/||b|| 2.604917963917e-12
>    21 SNES Function norm 2.752608806757e+02
>       0 KSP preconditioned resid norm 1.727563200247e+02 true resid norm
> 2.752608806757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.663026698374e-10 true resid norm
> 5.807801532466e-09 ||r(i)||/||b|| 2.109926233692e-11
>    22 SNES Function norm 2.707559177016e+02
>       0 KSP preconditioned resid norm 4.125046548626e+02 true resid norm
> 2.707559177016e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.109007703208e-10 true resid norm
> 5.875389803064e-09 ||r(i)||/||b|| 2.169994972940e-11
>    23 SNES Function norm 2.698855074973e+02
>       0 KSP preconditioned resid norm 7.414116220947e+01 true resid norm
> 2.698855074973e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.222534270039e-11 true resid norm
> 6.928234778488e-10 ||r(i)||/||b|| 2.567101450810e-12
>    24 SNES Function norm 2.616892369202e+02
>       0 KSP preconditioned resid norm 1.819161466540e+02 true resid norm
> 2.616892369202e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.013662651542e-09 true resid norm
> 4.195842561304e-09 ||r(i)||/||b|| 1.603368411588e-11
>    25 SNES Function norm 2.600248325147e+02
>       0 KSP preconditioned resid norm 1.265123744997e+02 true resid norm
> 2.600248325147e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.572744639399e-09 true resid norm
> 7.386966838391e-09 ||r(i)||/||b|| 2.840869761150e-11
>    26 SNES Function norm 2.593065715720e+02
>       0 KSP preconditioned resid norm 1.314965938655e+02 true resid norm
> 2.593065715720e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.156889834113e-10 true resid norm
> 3.157778817609e-09 ||r(i)||/||b|| 1.217778168314e-11
>    27 SNES Function norm 2.517348459439e+02
>       0 KSP preconditioned resid norm 2.618005944700e+01 true resid norm
> 2.517348459439e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.483924363849e-11 true resid norm
> 7.418968110702e-10 ||r(i)||/||b|| 2.947135936976e-12
>    28 SNES Function norm 2.298635352641e+02
>       0 KSP preconditioned resid norm 8.770042104744e+01 true resid norm
> 2.298635352641e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.098440763657e-09 true resid norm
> 3.936819814733e-09 ||r(i)||/||b|| 1.712676962968e-11
>    29 SNES Function norm 2.269871118080e+02
>       0 KSP preconditioned resid norm 2.285655807501e+02 true resid norm
> 2.269871118080e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.216116049541e-08 true resid norm
> 9.327420966977e-08 ||r(i)||/||b|| 4.109229327024e-10
>    30 SNES Function norm 2.252105956612e+02
>       0 KSP preconditioned resid norm 8.597156834231e+01 true resid norm
> 2.252105956612e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.431615405953e-09 true resid norm
> 4.297181487549e-09 ||r(i)||/||b|| 1.908072519826e-11
>    31 SNES Function norm 2.251883212710e+02
>       0 KSP preconditioned resid norm 7.066983917118e+01 true resid norm
> 2.251883212710e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.998573955346e-10 true resid norm
> 3.454505271056e-09 ||r(i)||/||b|| 1.534051700176e-11
>    32 SNES Function norm 2.117895362999e+02
>       0 KSP preconditioned resid norm 3.830985324064e+02 true resid norm
> 2.117895362999e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.016573825110e-09 true resid norm
> 1.942657883557e-08 ||r(i)||/||b|| 9.172586698552e-11
>    33 SNES Function norm 2.116404150058e+02
>       0 KSP preconditioned resid norm 5.329232125733e+02 true resid norm
> 2.116404150058e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.609537995048e-09 true resid norm
> 1.451505196923e-08 ||r(i)||/||b|| 6.858355465253e-11
>    34 SNES Function norm 2.115566314991e+02
>       0 KSP preconditioned resid norm 6.609724236205e+02 true resid norm
> 2.115566314991e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.254105497370e-10 true resid norm
> 1.178528765891e-08 ||r(i)||/||b|| 5.570748397436e-11
>    35 SNES Function norm 2.115028944726e+02
>       0 KSP preconditioned resid norm 8.217040556045e+02 true resid norm
> 2.115028944726e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.803324342385e-09 true resid norm
> 2.957366914143e-08 ||r(i)||/||b|| 1.398263093050e-10
>    36 SNES Function norm 2.114773510911e+02
>       0 KSP preconditioned resid norm 1.091957913212e+03 true resid norm
> 2.114773510911e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.264170340401e-08 true resid norm
> 1.635833495717e-07 ||r(i)||/||b|| 7.735265678696e-10
>    37 SNES Function norm 2.114573155596e+02
>       0 KSP preconditioned resid norm 1.295967660499e+03 true resid norm
> 2.114573155596e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.787403648651e-08 true resid norm
> 1.913077973453e-07 ||r(i)||/||b|| 9.047111793648e-10
>    38 SNES Function norm 2.114434864633e+02
>       0 KSP preconditioned resid norm 1.567323979910e+03 true resid norm
> 2.114434864633e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.869044911030e-09 true resid norm
> 3.425540409553e-08 ||r(i)||/||b|| 1.620073744928e-10
>    39 SNES Function norm 2.114337906450e+02
>       0 KSP preconditioned resid norm 1.835099219771e+03 true resid norm
> 2.114337906450e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.224665285731e-10 true resid norm
> 1.690151657152e-08 ||r(i)||/||b|| 7.993763210677e-11
>    40 SNES Function norm 2.114267348757e+02
>       0 KSP preconditioned resid norm 2.146249211809e+03 true resid norm
> 2.114267348757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.809299780773e-08 true resid norm
> 1.157600089098e-07 ||r(i)||/||b|| 5.475183116166e-10
>    41 SNES Function norm 2.114215910347e+02
>       0 KSP preconditioned resid norm 2.510692990698e+03 true resid norm
> 2.114215910347e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.485943641401e-08 true resid norm
> 9.439839396991e-08 ||r(i)||/||b|| 4.464936315535e-10
>    42 SNES Function norm 2.114206676373e+02
>       0 KSP preconditioned resid norm 3.401544811763e+03 true resid norm
> 2.114206676373e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.910138749741e-09 true resid norm
> 3.806926432018e-08 ||r(i)||/||b|| 1.800640625423e-10
>    43 SNES Function norm 2.114186309697e+02
>       0 KSP preconditioned resid norm 3.955962524189e+03 true resid norm
> 2.114186309697e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.500201400572e-08 true resid norm
> 1.570197988244e-07 ||r(i)||/||b|| 7.426961290224e-10
>    44 SNES Function norm 2.114171338084e+02
>       0 KSP preconditioned resid norm 4.636713819556e+03 true resid norm
> 2.114171338084e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.248004040828e-09 true resid norm
> 7.296845022145e-08 ||r(i)||/||b|| 3.451397193171e-10
>    45 SNES Function norm 2.114170015592e+02
>       0 KSP preconditioned resid norm 6.321621979244e+03 true resid norm
> 2.114170015592e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.674310076597e-08 true resid norm
> 1.788726595879e-07 ||r(i)||/||b|| 8.460656345928e-10
>    46 SNES Function norm 2.114164190412e+02
>       0 KSP preconditioned resid norm 7.414534654352e+03 true resid norm
> 2.114164190412e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.520165386851e-08 true resid norm
> 4.326674448851e-07 ||r(i)||/||b|| 2.046517705897e-09
>    47 SNES Function norm 2.114162562951e+02
>       0 KSP preconditioned resid norm 9.810108530473e+03 true resid norm
> 2.114162562951e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.690981568660e-08 true resid norm
> 1.826808079437e-07 ||r(i)||/||b|| 8.640811787375e-10
>    48 SNES Function norm 2.114160137794e+02
>       0 KSP preconditioned resid norm 1.136099973228e+04 true resid norm
> 2.114160137794e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.015775310105e-08 true resid norm
> 1.261980904678e-07 ||r(i)||/||b|| 5.969183138583e-10
>    49 SNES Function norm 2.114158331092e+02
>       0 KSP preconditioned resid norm 1.315560026827e+04 true resid norm
> 2.114158331092e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.131165057283e-08 true resid norm
> 3.097361970470e-07 ||r(i)||/||b|| 1.465056767470e-09
>    50 SNES Function norm 2.114158223286e+02
>       0 KSP preconditioned resid norm 1.794717411653e+04 true resid norm
> 2.114158223286e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.070352092082e-07 true resid norm
> 6.224398071454e-07 ||r(i)||/||b|| 2.944149592446e-09
>    51 SNES Function norm 2.114157500757e+02
>       0 KSP preconditioned resid norm 2.077631649185e+04 true resid norm
> 2.114157500757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.358541701826e-08 true resid norm
> 5.604873460299e-07 ||r(i)||/||b|| 2.651114431301e-09
>    52 SNES Function norm 2.114156961871e+02
>       0 KSP preconditioned resid norm 2.405048247398e+04 true resid norm
> 2.114156961871e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.439555573169e-08 true resid norm
> 3.896791261159e-07 ||r(i)||/||b|| 1.843189191455e-09
>    53 SNES Function norm 2.114156592651e+02
>       0 KSP preconditioned resid norm 2.914886710376e+04 true resid norm
> 2.114156592651e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.436687722935e-07 true resid norm
> 1.370484810325e-06 ||r(i)||/||b|| 6.482418639609e-09
>    54 SNES Function norm 2.114156319126e+02
>       0 KSP preconditioned resid norm 3.373835363786e+04 true resid norm
> 2.114156319126e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.028533154395e-07 true resid norm
> 6.668879459515e-07 ||r(i)||/||b|| 3.154392794508e-09
>    55 SNES Function norm 2.114156115019e+02
>       0 KSP preconditioned resid norm 3.904963797200e+04 true resid norm
> 2.114156115019e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.986152192528e-08 true resid norm
> 4.508900491313e-07 ||r(i)||/||b|| 2.132718799374e-09
>    56 SNES Function norm 2.114155964342e+02
>       0 KSP preconditioned resid norm 4.594596937760e+04 true resid norm
> 2.114155964342e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.567731897537e-08 true resid norm
> 5.054454305445e-07 ||r(i)||/||b|| 2.390766996709e-09
>    57 SNES Function norm 2.114155854345e+02
>       0 KSP preconditioned resid norm 5.317614782539e+04 true resid norm
> 2.114155854345e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.460625544912e-07 true resid norm
> 9.171801951021e-07 ||r(i)||/||b|| 4.338280894556e-09
>    58 SNES Function norm 2.114155777135e+02
>       0 KSP preconditioned resid norm 6.400017583847e+04 true resid norm
> 2.114155777135e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.093685259962e-08 true resid norm
> 6.506435082837e-07 ||r(i)||/||b|| 3.077557081273e-09
>    59 SNES Function norm 2.114155720468e+02
>       0 KSP preconditioned resid norm 7.406807436961e+04 true resid norm
> 2.114155720468e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.571128588743e-07 true resid norm
> 1.516640238741e-06 ||r(i)||/||b|| 7.173739493538e-09
>    60 SNES Function norm 2.114155680388e+02
>       0 KSP preconditioned resid norm 8.892466432276e+04 true resid norm
> 2.114155680388e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.335222631846e-08 true resid norm
> 8.161361513986e-07 ||r(i)||/||b|| 3.860340839464e-09
>    61 SNES Function norm 2.114155651044e+02
>       0 KSP preconditioned resid norm 1.029106627628e+05 true resid norm
> 2.114155651044e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.244108037270e-07 true resid norm
> 3.009616926997e-06 ||r(i)||/||b|| 1.423555037450e-08
>    62 SNES Function norm 2.114155629136e+02
>       0 KSP preconditioned resid norm 1.190953464375e+05 true resid norm
> 2.114155629136e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.108162336627e-08 true resid norm
> 9.193387836453e-07 ||r(i)||/||b|| 4.348491525295e-09
>    63 SNES Function norm 2.114155612779e+02
>       0 KSP preconditioned resid norm 1.378243866677e+05 true resid norm
> 2.114155612779e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.138603793608e-07 true resid norm
> 2.482322226118e-06 ||r(i)||/||b|| 1.174143573497e-08
>    64 SNES Function norm 2.114155602404e+02
>       0 KSP preconditioned resid norm 1.698559088929e+05 true resid norm
> 2.114155602404e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.608494949489e-08 true resid norm
> 1.125534374356e-06 ||r(i)||/||b|| 5.323801015765e-09
>    65 SNES Function norm 2.114155594364e+02
>       0 KSP preconditioned resid norm 1.965648734846e+05 true resid norm
> 2.114155594364e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.689853079655e-07 true resid norm
> 2.539748843840e-06 ||r(i)||/||b|| 1.201306493529e-08
>    66 SNES Function norm 2.114155588361e+02
>       0 KSP preconditioned resid norm 2.274726625349e+05 true resid norm
> 2.114155588361e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.937782777025e-07 true resid norm
> 5.126870538363e-06 ||r(i)||/||b|| 2.425020450996e-08
>    67 SNES Function norm 2.114155583879e+02
>       0 KSP preconditioned resid norm 2.632394448333e+05 true resid norm
> 2.114155583879e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.229084794159e-07 true resid norm
> 3.026195587359e-06 ||r(i)||/||b|| 1.431396823599e-08
>    68 SNES Function norm 2.114155581765e+02
>       0 KSP preconditioned resid norm 3.367643572924e+05 true resid norm
> 2.114155581765e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.632794900620e-08 true resid norm
> 2.446819364813e-06 ||r(i)||/||b|| 1.157350663271e-08
>    69 SNES Function norm 2.114155579720e+02
>       0 KSP preconditioned resid norm 3.897127303190e+05 true resid norm
> 2.114155579720e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.933486330809e-06 true resid norm
> 2.730183810468e-05 ||r(i)||/||b|| 1.291382638372e-07
>    70 SNES Function norm 2.114155578193e+02
>       0 KSP preconditioned resid norm 4.509848215072e+05 true resid norm
> 2.114155578193e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.706389710731e-08 true resid norm
> 3.785217910694e-06 ||r(i)||/||b|| 1.790415970205e-08
>    71 SNES Function norm 2.114155577054e+02
>       0 KSP preconditioned resid norm 5.242625554614e+05 true resid norm
> 2.114155577054e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.321749665453e-06 true resid norm
> 9.069981074375e-06 ||r(i)||/||b|| 4.290119976419e-08
>    72 SNES Function norm 2.114155576210e+02
>       0 KSP preconditioned resid norm 6.066871925199e+05 true resid norm
> 2.114155576210e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.327204488687e-07 true resid norm
> 7.090997425424e-06 ||r(i)||/||b|| 3.354056581842e-08
>    73 SNES Function norm 2.114155575580e+02
>       0 KSP preconditioned resid norm 7.020695777561e+05 true resid norm
> 2.114155575580e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.123816847857e-06 true resid norm
> 8.125899637337e-06 ||r(i)||/||b|| 3.843567489166e-08
>    74 SNES Function norm 2.114155575109e+02
>       0 KSP preconditioned resid norm 8.124474305522e+05 true resid norm
> 2.114155575109e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.293550435673e-07 true resid norm
> 7.844829319667e-06 ||r(i)||/||b|| 3.710620643072e-08
>    75 SNES Function norm 2.114155574865e+02
>       0 KSP preconditioned resid norm 1.029314742970e+06 true resid norm
> 2.114155574865e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.033392384072e-07 true resid norm
> 9.399925240813e-06 ||r(i)||/||b|| 4.446184260312e-08
>    76 SNES Function norm 2.114155574646e+02
>       0 KSP preconditioned resid norm 1.191137352032e+06 true resid norm
> 2.114155574646e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.636526184597e-07 true resid norm
> 1.077765913224e-05 ||r(i)||/||b|| 5.097855267366e-08
>    77 SNES Function norm 2.114155574575e+02
>       0 KSP preconditioned resid norm 1.563965951801e+06 true resid norm
> 2.114155574575e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.137216998536e-06 true resid norm
> 5.108848518118e-05 ||r(i)||/||b|| 2.416496013612e-07
>    78 SNES Function norm 2.114155574558e+02
>       0 KSP preconditioned resid norm 2.108895892585e+06 true resid norm
> 2.114155574558e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.138935971807e-05 true resid norm
> 6.467637231732e-05 ||r(i)||/||b|| 3.059205911601e-07
>    79 SNES Function norm 2.114155574511e+02
>       0 KSP preconditioned resid norm 2.564542138115e+06 true resid norm
> 2.114155574511e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.391129540980e-06 true resid norm
> 4.283623913226e-05 ||r(i)||/||b|| 2.026163052933e-07
>    80 SNES Function norm 2.114155574475e+02
>       0 KSP preconditioned resid norm 2.967732587697e+06 true resid norm
> 2.114155574475e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.446508619620e-06 true resid norm
> 5.164837794317e-05 ||r(i)||/||b|| 2.442979058246e-07
>    81 SNES Function norm 2.114155574449e+02
>       0 KSP preconditioned resid norm 3.434290246612e+06 true resid norm
> 2.114155574449e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.781979196562e-06 true resid norm
> 4.886329177979e-05 ||r(i)||/||b|| 2.311243901364e-07
>    82 SNES Function norm 2.114155574429e+02
>       0 KSP preconditioned resid norm 3.974205574181e+06 true resid norm
> 2.114155574429e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.425678262963e-05 true resid norm
> 8.517034218173e-05 ||r(i)||/||b|| 4.028574964485e-07
>    83 SNES Function norm 2.114155574416e+02
>       0 KSP preconditioned resid norm 4.785116089506e+06 true resid norm
> 2.114155574416e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.130667094145e-05 true resid norm
> 1.264608296603e-04 ||r(i)||/||b|| 5.981623641639e-07
>    84 SNES Function norm 2.114155574406e+02
>       0 KSP preconditioned resid norm 5.537350742612e+06 true resid norm
> 2.114155574406e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.299149833543e-06 true resid norm
> 5.419308321331e-05 ||r(i)||/||b|| 2.563344148812e-07
>    85 SNES Function norm 2.114155574399e+02
>       0 KSP preconditioned resid norm 6.767161611151e+06 true resid norm
> 2.114155574399e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.781986133631e-05 true resid norm
> 1.232616969009e-04 ||r(i)||/||b|| 5.830303994346e-07
>    86 SNES Function norm 2.114155574394e+02
>       0 KSP preconditioned resid norm 7.867884017146e+06 true resid norm
> 2.114155574394e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.002696664504e-05 true resid norm
> 1.398533205346e-04 ||r(i)||/||b|| 6.615091255747e-07
>    87 SNES Function norm 2.114155574390e+02
>       0 KSP preconditioned resid norm 9.102289060164e+06 true resid norm
> 2.114155574390e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.231201322143e-05 true resid norm
> 1.368407286352e-04 ||r(i)||/||b|| 6.472595030036e-07
>    88 SNES Function norm 2.114155574387e+02
>       0 KSP preconditioned resid norm 1.063720954841e+07 true resid norm
> 2.114155574387e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.517853677131e-05 true resid norm
> 1.273061147947e-04 ||r(i)||/||b|| 6.021605805032e-07
>    89 SNES Function norm 2.114155574385e+02
>       0 KSP preconditioned resid norm 1.230866787650e+07 true resid norm
> 2.114155574385e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.745229646688e-05 true resid norm
> 1.389443348166e-04 ||r(i)||/||b|| 6.572096041560e-07
>    90 SNES Function norm 2.114155574384e+02
>       0 KSP preconditioned resid norm 1.436772950622e+07 true resid norm
> 2.114155574384e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.955301916175e-05 true resid norm
> 5.633754040143e-04 ||r(i)||/||b|| 2.664777421494e-06
>    91 SNES Function norm 2.114155574383e+02
>       0 KSP preconditioned resid norm 1.662010558966e+07 true resid norm
> 2.114155574383e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.147116976734e-05 true resid norm
> 1.351152564949e-04 ||r(i)||/||b|| 6.390979837628e-07
>    92 SNES Function norm 2.114155574382e+02
>       0 KSP preconditioned resid norm 1.923255924422e+07 true resid norm
> 2.114155574382e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.030030646485e-05 true resid norm
> 1.877965090103e-04 ||r(i)||/||b|| 8.882814078868e-07
>    93 SNES Function norm 2.114155574381e+02
>       0 KSP preconditioned resid norm 2.226793244995e+07 true resid norm
> 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.411985876631e-04 true resid norm
> 1.342893408929e-03 ||r(i)||/||b|| 6.351913857247e-06
>    94 SNES Function norm 2.114155574381e+02
>       0 KSP preconditioned resid norm 2.576953439486e+07 true resid norm
> 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.147595096922e-05 true resid norm
> 4.578229507611e-04 ||r(i)||/||b|| 2.165512114193e-06
>    95 SNES Function norm 2.114155574380e+02
>       0 KSP preconditioned resid norm 2.981662207354e+07 true resid norm
> 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.456246888386e-06 true resid norm
> 2.856742906943e-04 ||r(i)||/||b|| 1.351245358460e-06
>    96 SNES Function norm 2.114155574380e+02
>       0 KSP preconditioned resid norm 3.451395888978e+07 true resid norm
> 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.972976681165e-05 true resid norm
> 3.126288732572e-04 ||r(i)||/||b|| 1.478741096662e-06
>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations
> 96
>
> -gideon
>
> On Sep 8, 2015, at 10:31 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Tue, Sep 8, 2015 at 9:19 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
>> Ok, going back and running the problem with the flags, -snes_fd
>> -snes_mf_operator -ksp_monitor_true_residual -snes_max_funcs 500000, during
>> the grid sequence, I still get:
>>
>
> Lets try using -pc_type lu instead of MUMPS, just to make sure. If we still
> get this, then the problem is really horribly conditioned without being
> singular.
> We are losing all precision for some reason. Is it possible that the two
> halves
> are so differently scaled that we make no progress on one with LU?
>
>   Thanks,
>
>      Matt
>
>
>>       0 SNES Function norm 9.131726454160e-09
>>         0 KSP preconditioned resid norm 4.201012306235e-09 true resid
>> norm 9.131726454160e-09 ||r(i)||/||b|| 1.000000000000e+00
>>         1 KSP preconditioned resid norm 4.811278613811e-14 true resid
>> norm 7.117495013042e-13 ||r(i)||/||b|| 7.794249037978e-05
>>         2 KSP preconditioned resid norm 1.729322442321e-19 true resid
>> norm 2.611605096203e-13 ||r(i)||/||b|| 2.859924800982e-05
>>       1 SNES Function norm 2.066497600915e-12
>>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>     0 SNES Function norm 5.066222213176e+03
>>       0 KSP preconditioned resid norm 3.135051257976e+01 true resid norm
>> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.425321207844e-04 true resid norm
>> 1.654961474093e+00 ||r(i)||/||b|| 3.266657885218e-04
>>       2 KSP preconditioned resid norm 2.739130995469e-09 true resid norm
>> 7.943827483388e-03 ||r(i)||/||b|| 1.567998234015e-06
>>     1 SNES Function norm 8.487402654330e+02
>>       0 KSP preconditioned resid norm 2.670376525819e+02 true resid norm
>> 8.487402654330e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.518628784729e-02 true resid norm
>> 1.820466086409e+00 ||r(i)||/||b|| 2.144903641964e-03
>>       2 KSP preconditioned resid norm 1.680654269556e-07 true resid norm
>> 1.635299027240e-01 ||r(i)||/||b|| 1.926736710678e-04
>>     2 SNES Function norm 6.547660268557e+02
>>       0 KSP preconditioned resid norm 1.529984375428e+02 true resid norm
>> 6.547660268557e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.106652647444e-02 true resid norm
>> 9.350438798847e-01 ||r(i)||/||b|| 1.428058026124e-03
>>       2 KSP preconditioned resid norm 1.739404692247e-07 true resid norm
>> 6.642022841611e-02 ||r(i)||/||b|| 1.014411647701e-04
>>     3 SNES Function norm 5.771035131885e+02
>>       0 KSP preconditioned resid norm 1.360697738776e+02 true resid norm
>> 5.771035131885e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.323138791370e-03 true resid norm
>> 6.419077863137e-02 ||r(i)||/||b|| 1.112292286642e-04
>>     4 SNES Function norm 5.238679723156e+02
>>       0 KSP preconditioned resid norm 1.528156345475e+02 true resid norm
>> 5.238679723156e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.115694692015e-02 true resid norm
>> 1.230231531838e+00 ||r(i)||/||b|| 2.348361795054e-03
>>       2 KSP preconditioned resid norm 5.444815701049e-08 true resid norm
>> 6.640979326953e-02 ||r(i)||/||b|| 1.267681873660e-04
>>     5 SNES Function norm 4.755828064476e+02
>>       0 KSP preconditioned resid norm 1.872910183528e+02 true resid norm
>> 4.755828064476e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.182180755863e-02 true resid norm
>> 5.918459188490e-01 ||r(i)||/||b|| 1.244464498769e-03
>>       2 KSP preconditioned resid norm 7.426717097996e-08 true resid norm
>> 8.473303784961e-02 ||r(i)||/||b|| 1.781667392111e-04
>>     6 SNES Function norm 4.224257767287e+02
>>       0 KSP preconditioned resid norm 2.578229460575e+02 true resid norm
>> 4.224257767287e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.258348180527e-02 true resid norm
>> 3.257118054248e-01 ||r(i)||/||b|| 7.710509712432e-04
>>       2 KSP preconditioned resid norm 1.122972268815e-07 true resid norm
>> 1.274729638407e-01 ||r(i)||/||b|| 3.017641698568e-04
>>     7 SNES Function norm 3.811223987907e+02
>>       0 KSP preconditioned resid norm 5.218684581634e+02 true resid norm
>> 3.811223987907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.089005298384e-01 true resid norm
>> 5.194667956652e+00 ||r(i)||/||b|| 1.362992039601e-02
>>       2 KSP preconditioned resid norm 7.570453480881e-07 true resid norm
>> 2.863356359719e-01 ||r(i)||/||b|| 7.512957435207e-04
>>     8 SNES Function norm 3.771290745624e+02
>>       0 KSP preconditioned resid norm 5.827455360446e+03 true resid norm
>> 3.771290745624e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.620225652579e-01 true resid norm
>> 1.332007773441e+01 ||r(i)||/||b|| 3.531967867995e-02
>>       2 KSP preconditioned resid norm 3.596801636560e-06 true resid norm
>> 3.425465452374e+00 ||r(i)||/||b|| 9.083005483862e-03
>>     9 SNES Function norm 3.767897776540e+02
>>       0 KSP preconditioned resid norm 5.185115189077e+03 true resid norm
>> 3.767897776540e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.021105455600e-01 true resid norm
>> 1.043813038739e+01 ||r(i)||/||b|| 2.770279611187e-02
>>       2 KSP preconditioned resid norm 6.951879108628e-06 true resid norm
>> 2.816562934830e+00 ||r(i)||/||b|| 7.475157506572e-03
>>    10 SNES Function norm 3.746968158051e+02
>>       0 KSP preconditioned resid norm 8.165048723911e+02 true resid norm
>> 3.746968158051e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.177528145230e-01 true resid norm
>> 2.394044923481e+00 ||r(i)||/||b|| 6.389285476945e-03
>>       2 KSP preconditioned resid norm 7.325770200602e-07 true resid norm
>> 4.877299135281e-01 ||r(i)||/||b|| 1.301665487816e-03
>>    11 SNES Function norm 3.726877791108e+02
>>       0 KSP preconditioned resid norm 5.024547551936e+02 true resid norm
>> 3.726877791108e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.640017871568e-01 true resid norm
>> 1.648101091752e+00 ||r(i)||/||b|| 4.422203206351e-03
>>       2 KSP preconditioned resid norm 1.005738633834e-06 true resid norm
>> 2.935508517682e-01 ||r(i)||/||b|| 7.876589151074e-04
>>    12 SNES Function norm 3.668832918639e+02
>>       0 KSP preconditioned resid norm 7.762104953873e+02 true resid norm
>> 3.668832918639e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.912259329681e-02 true resid norm
>> 1.334600681831e+00 ||r(i)||/||b|| 3.637670919956e-03
>>       2 KSP preconditioned resid norm 7.640320635302e-07 true resid norm
>> 4.471334143387e-01 ||r(i)||/||b|| 1.218734742776e-03
>>    13 SNES Function norm 3.631997889667e+02
>>       0 KSP preconditioned resid norm 2.515761606628e+02 true resid norm
>> 3.631997889667e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.648297544435e-01 true resid norm
>> 2.866637955219e+00 ||r(i)||/||b|| 7.892730233611e-03
>>       2 KSP preconditioned resid norm 7.305731711999e-07 true resid norm
>> 1.472589913546e-01 ||r(i)||/||b|| 4.054490003243e-04
>>    14 SNES Function norm 3.557844076846e+02
>>       0 KSP preconditioned resid norm 6.995145168850e+02 true resid norm
>> 3.557844076846e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.626527197886e-02 true resid norm
>> 5.642316830909e-01 ||r(i)||/||b|| 1.585880861848e-03
>>       2 KSP preconditioned resid norm 1.040702263139e-06 true resid norm
>> 4.073703976052e-01 ||r(i)||/||b|| 1.144992272866e-03
>>    15 SNES Function norm 3.534147071188e+02
>>       0 KSP preconditioned resid norm 2.698507093385e+02 true resid norm
>> 3.534147071188e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.592052963868e-01 true resid norm
>> 1.332346399928e+00 ||r(i)||/||b|| 3.769923472595e-03
>>       2 KSP preconditioned resid norm 5.021635239169e-07 true resid norm
>> 1.573071613558e-01 ||r(i)||/||b|| 4.451064378114e-04
>>    16 SNES Function norm 3.499900270010e+02
>>       0 KSP preconditioned resid norm 7.295084253442e+01 true resid norm
>> 3.499900270010e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.040248007780e-02 true resid norm
>> 1.427506717563e-01 ||r(i)||/||b|| 4.078706841436e-04
>>       2 KSP preconditioned resid norm 1.246491224109e-07 true resid norm
>> 4.074099442269e-02 ||r(i)||/||b|| 1.164061581177e-04
>>    17 SNES Function norm 3.473017914657e+02
>>       0 KSP preconditioned resid norm 4.169372990718e+02 true resid norm
>> 3.473017914657e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.140552389441e-01 true resid norm
>> 1.893314948416e+00 ||r(i)||/||b|| 5.451497789361e-03
>>       2 KSP preconditioned resid norm 7.751532205539e-07 true resid norm
>> 2.494992549211e-01 ||r(i)||/||b|| 7.183932275966e-04
>>    18 SNES Function norm 3.464416771335e+02
>>       0 KSP preconditioned resid norm 2.968698304201e+02 true resid norm
>> 3.464416771335e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.683113115222e-02 true resid norm
>> 2.501212385243e-01 ||r(i)||/||b|| 7.219721385540e-04
>>       2 KSP preconditioned resid norm 5.274081741342e-07 true resid norm
>> 1.712806350336e-01 ||r(i)||/||b|| 4.943996243489e-04
>>    19 SNES Function norm 3.460747441983e+02
>>       0 KSP preconditioned resid norm 3.707198448214e+02 true resid norm
>> 3.460747441983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.740664131652e-02 true resid norm
>> 9.477346570578e-01 ||r(i)||/||b|| 2.738525919461e-03
>>       2 KSP preconditioned resid norm 4.649501179917e-07 true resid norm
>> 2.217443031583e-01 ||r(i)||/||b|| 6.407410736429e-04
>>    20 SNES Function norm 3.455596398964e+02
>>       0 KSP preconditioned resid norm 5.000331867417e+02 true resid norm
>> 3.455596398964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.821124737573e-02 true resid norm
>> 7.651194558377e-01 ||r(i)||/||b|| 2.214145888296e-03
>>       2 KSP preconditioned resid norm 5.030283955029e-07 true resid norm
>> 2.918161448112e-01 ||r(i)||/||b|| 8.444740389784e-04
>>    21 SNES Function norm 3.453222949611e+02
>>       0 KSP preconditioned resid norm 7.399233760260e+02 true resid norm
>> 3.453222949611e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.030913689264e-01 true resid norm
>> 8.930798988128e-01 ||r(i)||/||b|| 2.586221370136e-03
>>       2 KSP preconditioned resid norm 1.497261432173e-06 true resid norm
>> 4.385255355661e-01 ||r(i)||/||b|| 1.269902181136e-03
>>    22 SNES Function norm 3.452850731057e+02
>>       0 KSP preconditioned resid norm 1.850020325701e+03 true resid norm
>> 3.452850731057e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.670597549481e-01 true resid norm
>> 6.205758169419e+00 ||r(i)||/||b|| 1.797285389027e-02
>>       2 KSP preconditioned resid norm 4.122440586032e-06 true resid norm
>> 1.109676174925e+00 ||r(i)||/||b|| 3.213797124053e-03
>>    23 SNES Function norm 3.452722963704e+02
>>       0 KSP preconditioned resid norm 3.362446711623e+03 true resid norm
>> 3.452722963704e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.181412468129e+00 true resid norm
>> 1.564006484993e+01 ||r(i)||/||b|| 4.529776936738e-02
>>       2 KSP preconditioned resid norm 1.165193864469e-05 true resid norm
>> 1.869411988347e+00 ||r(i)||/||b|| 5.414312147250e-03
>>    24 SNES Function norm 3.452679622612e+02
>>       0 KSP preconditioned resid norm 6.149140486077e+03 true resid norm
>> 3.452679622612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.400761945257e-01 true resid norm
>> 3.795886928810e+00 ||r(i)||/||b|| 1.099403171945e-02
>>       2 KSP preconditioned resid norm 1.754461070240e-06 true resid norm
>> 3.648059240080e+00 ||r(i)||/||b|| 1.056587821294e-02
>>    25 SNES Function norm 3.452667776779e+02
>>       0 KSP preconditioned resid norm 1.101683177789e+04 true resid norm
>> 3.452667776779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.317993935110e+00 true resid norm
>> 2.454711939414e+01 ||r(i)||/||b|| 7.109609432809e-02
>>       2 KSP preconditioned resid norm 3.457477291813e-05 true resid norm
>> 6.807469339003e+00 ||r(i)||/||b|| 1.971654899665e-02
>>    26 SNES Function norm 3.452666586474e+02
>>       0 KSP preconditioned resid norm 3.879190850784e+04 true resid norm
>> 3.452666586474e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.739562350939e+01 true resid norm
>> 1.221587956319e+02 ||r(i)||/||b|| 3.538099974971e-01
>>       2 KSP preconditioned resid norm 7.455463420492e-05 true resid norm
>> 3.080916578781e+01 ||r(i)||/||b|| 8.923295955800e-02
>>    27 SNES Function norm 3.452666425264e+02
>>       0 KSP preconditioned resid norm 5.709318799171e+04 true resid norm
>> 3.452666425264e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.292899448408e-01 true resid norm
>> 2.801354737138e+01 ||r(i)||/||b|| 8.113597990933e-02
>>    28 SNES Function norm 3.452666374187e+02
>>       0 KSP preconditioned resid norm 2.488913348294e+05 true resid norm
>> 3.452666374187e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.691998534170e+02 true resid norm
>> 7.244072115265e+03 ||r(i)||/||b|| 2.098109498625e+01
>>       2 KSP preconditioned resid norm 2.503929513070e-03 true resid norm
>> 2.928791518636e+03 ||r(i)||/||b|| 8.482694825461e+00
>>    29 SNES Function norm 3.452666374177e+02
>>       0 KSP preconditioned resid norm 2.533150641403e+05 true resid norm
>> 3.452666374177e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.817060595281e+02 true resid norm
>> 1.333479168158e+03 ||r(i)||/||b|| 3.862172082803e+00
>>       2 KSP preconditioned resid norm 8.930133492283e-04 true resid norm
>> 5.412764197416e+02 ||r(i)||/||b|| 1.567705538507e+00
>>    30 SNES Function norm 3.452666373128e+02
>>       0 KSP preconditioned resid norm 2.169409532487e+05 true resid norm
>> 3.452666373128e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.325438416139e+02 true resid norm
>> 9.441618139691e+02 ||r(i)||/||b|| 2.734587451940e+00
>>       2 KSP preconditioned resid norm 8.796591247860e-04 true resid norm
>> 2.978002325996e+02 ||r(i)||/||b|| 8.625224693509e-01
>>    31 SNES Function norm 3.452666372449e+02
>>       0 KSP preconditioned resid norm 1.459475274582e+05 true resid norm
>> 3.452666372449e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.978987062939e+02 true resid norm
>> 1.411576222768e+03 ||r(i)||/||b|| 4.088365542735e+00
>>       2 KSP preconditioned resid norm 1.235854958341e-03 true resid norm
>> 3.579390387115e+02 ||r(i)||/||b|| 1.036703231936e+00
>>    32 SNES Function norm 3.452666369928e+02
>>       0 KSP preconditioned resid norm 1.264234515603e+05 true resid norm
>> 3.452666369928e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.676975931928e+01 true resid norm
>> 1.855746501202e+02 ||r(i)||/||b|| 5.374821376792e-01
>>       2 KSP preconditioned resid norm 9.286140031498e-05 true resid norm
>> 3.495715551553e+01 ||r(i)||/||b|| 1.012468387331e-01
>>    33 SNES Function norm 3.452666245901e+02
>>       0 KSP preconditioned resid norm 1.544591438108e+06 true resid norm
>> 3.452666245901e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.501087626485e+01 true resid norm
>> 3.233291920369e+02 ||r(i)||/||b|| 9.364623424597e-01
>>       2 KSP preconditioned resid norm 1.867215855867e-04 true resid norm
>> 7.114686634655e+01 ||r(i)||/||b|| 2.060635499623e-01
>>    34 SNES Function norm 3.452666216061e+02
>>       0 KSP preconditioned resid norm 3.431570281248e+05 true resid norm
>> 3.452666216061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.140176766342e+02 true resid norm
>> 1.760366951993e+03 ||r(i)||/||b|| 5.098572644538e+00
>>       2 KSP preconditioned resid norm 2.324744486242e-03 true resid norm
>> 1.080605579953e+03 ||r(i)||/||b|| 3.129771348664e+00
>>    35 SNES Function norm 3.452666215921e+02
>>       0 KSP preconditioned resid norm 3.734063885343e+05 true resid norm
>> 3.452666215921e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.279219133674e+00 true resid norm
>> 1.691481367261e+02 ||r(i)||/||b|| 4.899058470990e-01
>>       2 KSP preconditioned resid norm 4.046466703016e-04 true resid norm
>> 1.964786369469e+02 ||r(i)||/||b|| 5.690635139908e-01
>>    36 SNES Function norm 3.452666212200e+02
>>       0 KSP preconditioned resid norm 7.952390829283e+05 true resid norm
>> 3.452666212200e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.707607129733e+01 true resid norm
>> 2.806640653978e+02 ||r(i)||/||b|| 8.128908158167e-01
>>       2 KSP preconditioned resid norm 1.296391885225e-04 true resid norm
>> 1.079623998255e+02 ||r(i)||/||b|| 3.126928384910e-01
>>    37 SNES Function norm 3.452666199636e+02
>>       0 KSP preconditioned resid norm 9.358626316710e+05 true resid norm
>> 3.452666199636e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.722696571742e+01 true resid norm
>> 3.251492717238e+02 ||r(i)||/||b|| 9.417338744130e-01
>>       2 KSP preconditioned resid norm 2.862108374145e-04 true resid norm
>> 3.320104402606e+01 ||r(i)||/||b|| 9.616059620694e-02
>>    38 SNES Function norm 3.452666133867e+02
>>       0 KSP preconditioned resid norm 8.365871848022e+04 true resid norm
>> 3.452666133867e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.482543031390e+02 true resid norm
>> 1.772891732407e+03 ||r(i)||/||b|| 5.134848443690e+00
>>       2 KSP preconditioned resid norm 1.439398367387e-03 true resid norm
>> 3.088902452162e+02 ||r(i)||/||b|| 8.946426710257e-01
>>    39 SNES Function norm 3.452666132332e+02
>>       0 KSP preconditioned resid norm 8.960112658041e+04 true resid norm
>> 3.452666132332e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.505891399150e+01 true resid norm
>> 1.787713293837e+02 ||r(i)||/||b|| 5.177776319280e-01
>>       2 KSP preconditioned resid norm 6.384856870578e-05 true resid norm
>> 2.543851208517e+01 ||r(i)||/||b|| 7.367787996341e-02
>>    40 SNES Function norm 3.452665904125e+02
>>       0 KSP preconditioned resid norm 7.796808144307e+05 true resid norm
>> 3.452665904125e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.020764189760e+01 true resid norm
>> 6.961545395677e+02 ||r(i)||/||b|| 2.016281212544e+00
>>       2 KSP preconditioned resid norm 3.506849883654e-04 true resid norm
>> 4.956460306859e+02 ||r(i)||/||b|| 1.435545878024e+00
>>    41 SNES Function norm 3.452665902941e+02
>>       0 KSP preconditioned resid norm 4.990797566798e+05 true resid norm
>> 3.452665902941e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.012324354359e+02 true resid norm
>> 7.390530765626e+02 ||r(i)||/||b|| 2.140528789458e+00
>>       2 KSP preconditioned resid norm 7.111862680292e-04 true resid norm
>> 4.066260265930e+02 ||r(i)||/||b|| 1.177716112777e+00
>>    42 SNES Function norm 3.452665901159e+02
>>       0 KSP preconditioned resid norm 3.472654025442e+05 true resid norm
>> 3.452665901159e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.552680791160e+02 true resid norm
>> 1.064105325830e+03 ||r(i)||/||b|| 3.081981739016e+00
>>       2 KSP preconditioned resid norm 2.695346315896e-03 true resid norm
>> 5.359619569435e+02 ||r(i)||/||b|| 1.552313407340e+00
>>    43 SNES Function norm 3.452665900083e+02
>>       0 KSP preconditioned resid norm 2.823873507449e+05 true resid norm
>> 3.452665900083e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.616335635707e+00 true resid norm
>> 1.564312339233e+02 ||r(i)||/||b|| 4.530737651723e-01
>>       2 KSP preconditioned resid norm 1.580836053583e-04 true resid norm
>> 1.777870114826e+02 ||r(i)||/||b|| 5.149267743465e-01
>>    44 SNES Function norm 3.452665895210e+02
>>       0 KSP preconditioned resid norm 4.764215602672e+05 true resid norm
>> 3.452665895210e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.491949190147e+01 true resid norm
>> 4.652947485976e+02 ||r(i)||/||b|| 1.347639078670e+00
>>       2 KSP preconditioned resid norm 2.266106888017e-04 true resid norm
>> 1.316722015429e+02 ||r(i)||/||b|| 3.813638664707e-01
>>    45 SNES Function norm 3.452665889075e+02
>>       0 KSP preconditioned resid norm 1.439659890301e+05 true resid norm
>> 3.452665889075e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.431461947162e+01 true resid norm
>> 1.403614551576e+02 ||r(i)||/||b|| 4.065306625865e-01
>>       2 KSP preconditioned resid norm 4.711117728059e-04 true resid norm
>> 1.095117712535e+02 ||r(i)||/||b|| 3.171803318704e-01
>>    46 SNES Function norm 3.452665876547e+02
>>       0 KSP preconditioned resid norm 2.157313334766e+05 true resid norm
>> 3.452665876547e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.967750174401e+00 true resid norm
>> 1.053591658730e+02 ||r(i)||/||b|| 3.051530893525e-01
>>       2 KSP preconditioned resid norm 4.512059841816e-04 true resid norm
>> 1.189061020519e+02 ||r(i)||/||b|| 3.443892525473e-01
>>    47 SNES Function norm 3.452665866229e+02
>>       0 KSP preconditioned resid norm 3.981118078251e+05 true resid norm
>> 3.452665866229e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.013119377232e+01 true resid norm
>> 4.969789311062e+02 ||r(i)||/||b|| 1.439406390196e+00
>>       2 KSP preconditioned resid norm 1.872910444347e-04 true resid norm
>> 1.396396719776e+02 ||r(i)||/||b|| 4.044401554852e-01
>>    48 SNES Function norm 3.452665850954e+02
>>       0 KSP preconditioned resid norm 1.967554630062e+05 true resid norm
>> 3.452665850954e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.006970327024e+00 true resid norm
>> 9.616098906643e+01 ||r(i)||/||b|| 2.785122951874e-01
>>       2 KSP preconditioned resid norm 3.184368801558e-05 true resid norm
>> 9.951979756268e+01 ||r(i)||/||b|| 2.882404549377e-01
>>    49 SNES Function norm 3.452665835600e+02
>>       0 KSP preconditioned resid norm 3.965648247310e+05 true resid norm
>> 3.452665835600e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.862969730098e+01 true resid norm
>> 1.790616593203e+02 ||r(i)||/||b|| 5.186185627174e-01
>>       2 KSP preconditioned resid norm 2.964457441143e-04 true resid norm
>> 1.447818294544e+02 ||r(i)||/||b|| 4.193334552146e-01
>>    50 SNES Function norm 3.452665828612e+02
>>       0 KSP preconditioned resid norm 1.265022286683e+06 true resid norm
>> 3.452665828612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.401066920177e+01 true resid norm
>> 3.143671803790e+02 ||r(i)||/||b|| 9.105056671685e-01
>>       2 KSP preconditioned resid norm 1.444898193175e-04 true resid norm
>> 7.578180059616e+01 ||r(i)||/||b|| 2.194877939480e-01
>>    51 SNES Function norm 3.452665817118e+02
>>       0 KSP preconditioned resid norm 2.063334812849e+05 true resid norm
>> 3.452665817118e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.074628697499e+01 true resid norm
>> 1.155569679685e+02 ||r(i)||/||b|| 3.346891187543e-01
>>       2 KSP preconditioned resid norm 1.326420665890e-04 true resid norm
>> 9.261090091148e+01 ||r(i)||/||b|| 2.682301323584e-01
>>    52 SNES Function norm 3.452665807811e+02
>>       0 KSP preconditioned resid norm 6.549963238965e+06 true resid norm
>> 3.452665807811e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.246352248726e+01 true resid norm
>> 3.604213542889e+02 ||r(i)||/||b|| 1.043892963731e+00
>>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations
>> 52
>>
>> which is a little different, but not so much different, than what i get
>> if I use my Jacobian.
>>
>> -gideon
>>
>> On Sep 8, 2015, at 10:00 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> Yup, now it?s working.
>> -gideon
>>
>> On Sep 8, 2015, at 9:58 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Tue, Sep 8, 2015 at 8:51 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>  wrote:
>>
>>> Going back to this point, a case that previously worked, is now giving
>>> me problems that I don?t understand:
>>>
>>>   0 SNES Function norm 1.421454390131e-02
>>>     0 KSP preconditioned resid norm 7.635339584553e-03 true resid norm
>>> 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP preconditioned resid norm 2.838956718317e-07 true resid norm
>>> 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
>>>     2 KSP preconditioned resid norm 5.889748199467e-12 true resid norm
>>> 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
>>>   1 SNES Function norm 4.476139939604e-06
>>>     0 KSP preconditioned resid norm 4.897102387844e-05 true resid norm
>>> 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP preconditioned resid norm 1.133649519960e-09 true resid norm
>>> 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
>>>     2 KSP preconditioned resid norm 2.223328042462e-14 true resid norm
>>> 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
>>>   2 SNES Function norm 5.447761863330e-09
>>>     0 KSP preconditioned resid norm 9.703890346186e-09 true resid norm
>>> 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP preconditioned resid norm 1.982949185218e-13 true resid norm
>>> 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
>>>     2 KSP preconditioned resid norm 7.045835741315e-18 true resid norm
>>> 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
>>> Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT
>>> iterations 2
>>>
>>
>> Somehow the maximum number of function evaluations is being exceeded. Oh,
>> are you using -snes_fd? This uses a function evaluation
>> for each column, so you might have to up it > 10000. The convergence is
>> fine.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> -gideon
>>>
>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>>   You can run with -snes_fd it will be slow so run with only two levels
>>> of grid sequencing; check the convergence rate of Newton on the finer grid.
>>> It will hopefully be much better (also KSP should converge in one
>>> iteration). Send the output
>>>
>>>  Barry
>>>
>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>> wrote:
>>>
>>> I went through my code and nothing jumps out at me as clearly wrong.  Is
>>> there a way to dump to disk vectors and matrices that would be useful for
>>> the comparison?  I imagine I?d want the jacobian that I am specifying, but
>>> what would I want to compare that against in terms of using
>>> -snes_mf_operator?
>>>
>>>
>>> -gideon
>>>
>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <
>>> gideon.simpson at gmail.com> wrote:
>>> Just to clarify, again, during the linear solve phase, even though the
>>> PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the
>>> true residual norm could be comparatively large signified there is
>>> something wrong.  The one thing i was curious about is why it seems that
>>> there is only an issue during grid sequencing.  I?m not seeing that kind of
>>> behavior on the coarsest mesh.
>>>
>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you
>>> assume a certain mesh size, and when
>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>
>>>   Matt
>>>
>>> -gideon
>>>
>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>> wrote:
>>>
>>> Alright, I?ll take another look at what has been implemented in my
>>> Jacobian file.  One thing I?m a bit unclear about is do I need these
>>>  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>
>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>
>>>
>>> Gideon,
>>>
>>>  In your case the DM creates that Jacobian matrix object and
>>> form_function_jacobian's job is just to fill it up with the correct values
>>> for the current solution value.   If you use -snes_mf_operator then SNES
>>> uses a matrix-free multiple for the first matrix and your Jacobian to build
>>> the preconditioner. With -snes_mf it uses a matrix-free multiple for the
>>> first and second matrix and ignores what you compute. Since it is matrix
>>> free in both it cannot build a preconditioner hence the -pc_type none
>>>
>>> Using -snes_mf and -snes_mf_operator are a way of testing if the
>>> Jacobian you provide is correct
>>>
>>>
>>> Barry
>>>
>>>
>>>
>>> -gideon
>>>
>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>> If the Jacobian you provide is correct then using -snes_mf with a PC
>>> type of LU should give two iterations (or at most three) to get very good
>>> (10^-12) convergence of the linear system. Here you are not getting that
>>> after a few SNES iterations. So I am strongly leaning to something wrong
>>> with the Jacobian you provide.  -snes_type test unfortunately doesn't catch
>>> all problems.
>>>
>>> You can try -snes_mf -pc_type none and see how the linear solver
>>> converges.
>>>
>>>
>>> Barry
>>>
>>>
>>>
>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>> wrote:
>>>
>>> Off the top of my head, I can?t rule out a null space in this problem. I
>>> thought I did a pretty good job of coding the Jacobian by hand.  And before
>>> I made that change to the Jacobian file, changing J for Jpre, if I ran with
>>>  -snes_check_jacobian, it appeared to be satisfactory.
>>>
>>> When I run with -ksp_type fgmres, I get
>>>
>>>   0 SNES Function norm 3.857327250267e-09
>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm
>>> 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm
>>> 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm
>>> 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>   1 SNES Function norm 2.479697868412e-12
>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>> 0 SNES Function norm 5.066222213176e+03
>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm
>>> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm
>>> 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>> 1 SNES Function norm 8.484696862318e+02
>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm
>>> 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm
>>> 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm
>>> 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>> 2 SNES Function norm 6.551139809777e+02
>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm
>>> 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm
>>> 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm
>>> 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>> 3 SNES Function norm 5.771483337231e+02
>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm
>>> 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm
>>> 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm
>>> 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>> 4 SNES Function norm 5.237540245670e+02
>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm
>>> 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm
>>> 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm
>>> 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>> 5 SNES Function norm 4.753058321803e+02
>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm
>>> 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm
>>> 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm
>>> 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>> 6 SNES Function norm 4.221482124411e+02
>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm
>>> 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm
>>> 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm
>>> 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>> 7 SNES Function norm 3.803620028964e+02
>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm
>>> 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm
>>> 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm
>>> 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>> 8 SNES Function norm 3.769519577570e+02
>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm
>>> 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm
>>> 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm
>>> 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm
>>> 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>> 9 SNES Function norm 3.766389203406e+02
>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm
>>> 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm
>>> 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm
>>> 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>> 10 SNES Function norm 3.751822489648e+02
>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm
>>> 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm
>>> 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm
>>> 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>> 11 SNES Function norm 3.737189963925e+02
>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm
>>> 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm
>>> 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm
>>> 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm
>>> 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>> 12 SNES Function norm 3.737110331594e+02
>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm
>>> 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm
>>> 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm
>>> 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>> 13 SNES Function norm 3.684175513154e+02
>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm
>>> 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm
>>> 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm
>>> 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>> 14 SNES Function norm 3.509930783933e+02
>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm
>>> 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm
>>> 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm
>>> 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>> 15 SNES Function norm 3.162121096565e+02
>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm
>>> 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm
>>> 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm
>>> 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>> 16 SNES Function norm 2.890165052818e+02
>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm
>>> 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm
>>> 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm
>>> 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>> 17 SNES Function norm 2.816929421445e+02
>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm
>>> 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm
>>> 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm
>>> 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm
>>> 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>> 18 SNES Function norm 2.814407559738e+02
>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm
>>> 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm
>>> 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm
>>> 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>> 19 SNES Function norm 2.810261821835e+02
>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm
>>> 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm
>>> 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm
>>> 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>> 20 SNES Function norm 2.755723672077e+02
>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm
>>> 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm
>>> 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm
>>> 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm
>>> 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm
>>> 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm
>>> 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm
>>> 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>> 21 SNES Function norm 2.755720389981e+02
>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm
>>> 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm
>>> 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm
>>> 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm
>>> 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm
>>> 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm
>>> 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm
>>> 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>> 22 SNES Function norm 2.755716464881e+02
>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm
>>> 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm
>>> 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm
>>> 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm
>>> 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm
>>> 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>> 23 SNES Function norm 2.755713163168e+02
>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm
>>> 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm
>>> 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm
>>> 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm
>>> 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm
>>> 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm
>>> 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm
>>> 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm
>>> 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm
>>> 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>> 24 SNES Function norm 2.755712363532e+02
>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm
>>> 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm
>>> 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm
>>> 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm
>>> 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm
>>> 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm
>>> 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm
>>> 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm
>>> 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm
>>> 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm
>>> 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm
>>> 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm
>>> 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm
>>> 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm
>>> 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm
>>> 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm
>>> 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm
>>> 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm
>>> 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm
>>> 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm
>>> 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm
>>> 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm
>>> 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm
>>> 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm
>>> 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm
>>> 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm
>>> 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm
>>> 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm
>>> 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm
>>> 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm
>>> 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm
>>> 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm
>>> 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm
>>> 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm
>>> 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm
>>> 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm
>>> 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm
>>> 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm
>>> 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm
>>> 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm
>>> 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm
>>> 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm
>>> 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm
>>> 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm
>>> 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm
>>> 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm
>>> 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm
>>> 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm
>>> 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm
>>> 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm
>>> 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm
>>> 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm
>>> 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm
>>> 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm
>>> 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm
>>> 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm
>>> 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm
>>> 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm
>>> 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm
>>> 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm
>>> 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm
>>> 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm
>>> 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm
>>> 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm
>>> 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm
>>> 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm
>>> 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm
>>> 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm
>>> 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm
>>> 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm
>>> 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm
>>> 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm
>>> 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm
>>> 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm
>>> 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm
>>> 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm
>>> 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm
>>> 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm
>>> 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm
>>> 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm
>>> 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm
>>> 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm
>>> 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm
>>> 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm
>>> 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm
>>> 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm
>>> 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm
>>> 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm
>>> 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm
>>> 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm
>>> 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm
>>> 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm
>>> 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm
>>> 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm
>>> 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm
>>> 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm
>>> 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm
>>> 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm
>>> 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm
>>> 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm
>>> 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm
>>> 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm
>>> 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm
>>> 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm
>>> 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm
>>> 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm
>>> 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm
>>> 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm
>>> 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm
>>> 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm
>>> 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm
>>> 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm
>>> 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm
>>> 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm
>>> 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm
>>> 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm
>>> 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm
>>> 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm
>>> 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm
>>> 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm
>>> 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm
>>> 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm
>>> 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm
>>> 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm
>>> 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm
>>> 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm
>>> 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm
>>> 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm
>>> 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm
>>> 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm
>>> 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm
>>> 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm
>>> 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm
>>> 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm
>>> 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm
>>> 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm
>>> 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm
>>> 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm
>>> 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm
>>> 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm
>>> 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm
>>> 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm
>>> 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm
>>> 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm
>>> 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm
>>> 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm
>>> 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm
>>> 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm
>>> 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm
>>> 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm
>>> 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm
>>> 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm
>>> 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm
>>> 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm
>>> 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm
>>> 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm
>>> 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm
>>> 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm
>>> 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm
>>> 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm
>>> 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm
>>> 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm
>>> 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm
>>> 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm
>>> 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm
>>> 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm
>>> 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm
>>> 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm
>>> 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm
>>> 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm
>>> 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm
>>> 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm
>>> 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm
>>> 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm
>>> 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm
>>> 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm
>>> 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm
>>> 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm
>>> 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm
>>> 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm
>>> 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm
>>> 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm
>>> 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm
>>> 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm
>>> 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm
>>> 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm
>>> 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm
>>> 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm
>>> 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm
>>> 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm
>>> 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm
>>> 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm
>>> 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm
>>> 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm
>>> 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm
>>> 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm
>>> 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm
>>> 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm
>>> 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm
>>> 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm
>>> 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm
>>> 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm
>>> 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm
>>> 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm
>>> 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm
>>> 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm
>>> 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm
>>> 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm
>>> 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm
>>> 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm
>>> 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm
>>> 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations
>>> 24
>>>
>>> -gideon
>>>
>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>> Is there any chance your Jacobian may have a null space? (For example
>>> with Neumann boundary conditions there is often a nullspace)
>>>
>>> If not run with with -ksp_type fgmres and send the same output.
>>>
>>> Something is screwy with the linear system convergence, could be several
>>> things including either a null space problem or that your Jacobian that you
>>> provide is terribly wrong.
>>>
>>> Barry
>>>
>>>
>>> 21 SNES Function norm 2.987064584431e+02
>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm
>>> 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm
>>> 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm
>>> 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>
>>>
>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>> wrote:
>>>
>>> Got it, fixing that, and returning to the original question, this is
>>> what I now get, when I use those two flags:
>>>
>>>  0 SNES Function norm 1.132185384796e-08
>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm
>>> 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm
>>> 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>  1 SNES Function norm 2.177599365111e-12
>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>> 0 SNES Function norm 5.066222213176e+03
>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm
>>> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm
>>> 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>> 1 SNES Function norm 8.482593852817e+02
>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm
>>> 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm
>>> 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm
>>> 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>> 2 SNES Function norm 6.543140468549e+02
>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm
>>> 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm
>>> 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm
>>> 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>> 3 SNES Function norm 5.766430557220e+02
>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm
>>> 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm
>>> 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm
>>> 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>> 4 SNES Function norm 5.235211958260e+02
>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm
>>> 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm
>>> 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm
>>> 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>> 5 SNES Function norm 4.752913229649e+02
>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm
>>> 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm
>>> 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm
>>> 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>> 6 SNES Function norm 4.220255380391e+02
>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm
>>> 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm
>>> 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm
>>> 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>> 7 SNES Function norm 3.805408907074e+02
>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm
>>> 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm
>>> 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>> 8 SNES Function norm 3.764619752339e+02
>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm
>>> 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm
>>> 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm
>>> 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>> 9 SNES Function norm 3.761182227091e+02
>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm
>>> 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm
>>> 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm
>>> 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>> 10 SNES Function norm 3.740017190063e+02
>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm
>>> 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm
>>> 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm
>>> 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>> 11 SNES Function norm 3.725903477238e+02
>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm
>>> 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm
>>> 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm
>>> 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>> 12 SNES Function norm 3.716162097231e+02
>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm
>>> 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm
>>> 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm
>>> 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>> 13 SNES Function norm 3.674168632847e+02
>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm
>>> 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm
>>> 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm
>>> 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>> 14 SNES Function norm 3.532395445266e+02
>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm
>>> 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm
>>> 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm
>>> 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>> 15 SNES Function norm 3.182438872366e+02
>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm
>>> 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm
>>> 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm
>>> 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>> 16 SNES Function norm 3.091759892779e+02
>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm
>>> 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm
>>> 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm
>>> 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>> 17 SNES Function norm 2.987839504359e+02
>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm
>>> 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm
>>> 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm
>>> 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>> 18 SNES Function norm 2.987073622777e+02
>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm
>>> 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm
>>> 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm
>>> 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>> 19 SNES Function norm 2.987067936734e+02
>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm
>>> 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm
>>> 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm
>>> 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>> 20 SNES Function norm 2.987067502910e+02
>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm
>>> 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm
>>> 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm
>>> 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>> 21 SNES Function norm 2.987064584431e+02
>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm
>>> 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm
>>> 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm
>>> 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>> 22 SNES Function norm 2.987064525262e+02
>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm
>>> 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm
>>> 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm
>>> 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>> 23 SNES Function norm 2.987064121622e+02
>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm
>>> 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm
>>> 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm
>>> 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>> 24 SNES Function norm 2.987063973426e+02
>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm
>>> 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm
>>> 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm
>>> 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>> 25 SNES Function norm 2.987063920553e+02
>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm
>>> 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm
>>> 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm
>>> 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>> 26 SNES Function norm 2.987063919786e+02
>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm
>>> 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm
>>> 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm
>>> 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>> 27 SNES Function norm 2.987063393145e+02
>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm
>>> 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm
>>> 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm
>>> 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>> 28 SNES Function norm 2.987063253277e+02
>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm
>>> 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm
>>> 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm
>>> 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>> 29 SNES Function norm 2.987063197153e+02
>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm
>>> 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm
>>> 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm
>>> 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>> 30 SNES Function norm 2.987063193089e+02
>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm
>>> 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm
>>> 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm
>>> 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>> 31 SNES Function norm 2.987063192061e+02
>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm
>>> 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm
>>> 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm
>>> 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>> 32 SNES Function norm 2.987063094492e+02
>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm
>>> 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm
>>> 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm
>>> 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>> 33 SNES Function norm 2.987063089942e+02
>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm
>>> 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm
>>> 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm
>>> 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>> 34 SNES Function norm 2.987062660109e+02
>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm
>>> 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm
>>> 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm
>>> 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>> 35 SNES Function norm 2.987062107900e+02
>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm
>>> 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm
>>> 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm
>>> 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>> 36 SNES Function norm 2.987062055224e+02
>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm
>>> 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm
>>> 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm
>>> 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>> 37 SNES Function norm 2.987061774798e+02
>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm
>>> 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm
>>> 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm
>>> 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>> 38 SNES Function norm 2.987061715400e+02
>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm
>>> 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm
>>> 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm
>>> 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>> 39 SNES Function norm 2.987061699634e+02
>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm
>>> 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm
>>> 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm
>>> 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>> 40 SNES Function norm 2.987061630064e+02
>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm
>>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm
>>> 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>> 41 SNES Function norm 2.987061630064e+02
>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm
>>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm
>>> 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>> 42 SNES Function norm 2.987061630064e+02
>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm
>>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm
>>> 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>> 43 SNES Function norm 2.987061630064e+02
>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm
>>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm
>>> 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>> 44 SNES Function norm 2.987061630064e+02
>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm
>>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm
>>> 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>> 45 SNES Function norm 2.987061630064e+02
>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm
>>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm
>>> 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>> 46 SNES Function norm 2.987061630064e+02
>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm
>>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm
>>> 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>> 47 SNES Function norm 2.987061630064e+02
>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm
>>> 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm
>>> 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations
>>> 47
>>>
>>>
>>> -gideon
>>>
>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>> This indicates that somewhere in your ComputeJacobian you are setting
>>> matrix entries with the first Mat argument when you should always set them
>>> with the second matrix argument. For example if you have
>>>
>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>
>>> you should call all the MatSetValues() with jpre, no J. Then at the end
>>> of the function you should call MatAssemblyBegin/End() on jpre then on J if
>>> J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>
>>> This is a minor glitch we'll get past.
>>>
>>> Barry
>>>
>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>> wrote:
>>>
>>> By the way, I tried using a different petsc installation, and now,
>>> rather than the segmentation fault, I get the following error:
>>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: No support for this operation for this object type
>>> [0]PETSC ERROR: Mat type mffd
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
>>> trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named
>>> gs_air by gideon Mon Sep  7 21:32:18 2015
>>> [0]PETSC ERROR: Configure options --download-mpich=yes
>>> --download-suitesparse=yes --download-superlu=yes
>>> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes
>>> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes
>>> --download-parmetis=yes --download-scalapack=yes
>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in
>>> /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>
>>> -gideon
>>>
>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>> Hmm,
>>>
>>> Ok you can try running it directly in the debugger since it is one
>>> process, type
>>>
>>> gdb ./blowup_batch_refine
>>>
>>> then
>>>
>>> when the debugger comes up (if it does not cut and paste all output and
>>> send it)
>>>
>>> run -on_error_abort -snes_mf_operator  and any other options you
>>> normally use
>>>
>>>
>>> Barry
>>>
>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>> wrote:
>>>
>>> Running with that flag gives me this:
>>>
>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid
>>> 16111 on gs_air
>>> Unable to start debugger: No such file or directory
>>>
>>>
>>>
>>> -gideon
>>>
>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>> This should not happen. Run with a debug version of PETSc installed and
>>> the option -start_in_debugger noxterm  Once the debugger starts up type
>>> cont and when it crashes type where or bt  Send all output
>>>
>>>
>>>
>>> Barry
>>>
>>>
>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>> wrote:
>>>
>>> I?m getting an error with -snes_mf_operator,
>>>
>>> 0 SNES Function norm 1.421454390131e-02
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>> probably memory access out of range
>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [0]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>> OS X to find memory corruption errors
>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
>>> and run
>>> [0]PETSC ERROR: to get more information on the crash.
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Signal received
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
>>> trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by
>>> gideon Mon Sep  7 21:08:19 2015
>>> [0]PETSC ERROR: Configure options --prefix=/opt/local
>>> --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries
>>> --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0
>>> --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate
>>> --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local
>>> --with-superlu-dir=/opt/local --with-metis-dir=/opt/local
>>> --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local
>>> --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local
>>> CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp
>>> FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp
>>> F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os
>>> FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names"
>>> CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os
>>> FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports
>>> --with-mpiexec=mpiexec-mpich-mp
>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>
>>> -gideon
>>>
>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>> My guess is the Jacobian is not correct (or correct "enough"), hence
>>> PETSc SNES is generating a poor descent direction. You can try
>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments.
>>> What happens?
>>>
>>> Barry
>>>
>>>
>>>
>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>> wrote:
>>>
>>> No problem Matt, I don?t think we had previously discussed that output.
>>> Here is a case where things fail.
>>>
>>> 0 SNES Function norm 4.027481756921e-09
>>> 1 SNES Function norm 1.760477878365e-12
>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>> 0 SNES Function norm 5.066222213176e+03
>>> 1 SNES Function norm 8.484697184230e+02
>>> 2 SNES Function norm 6.549559723294e+02
>>> 3 SNES Function norm 5.770723278153e+02
>>> 4 SNES Function norm 5.237702240594e+02
>>> 5 SNES Function norm 4.753909019848e+02
>>> 6 SNES Function norm 4.221784590755e+02
>>> 7 SNES Function norm 3.806525080483e+02
>>> 8 SNES Function norm 3.762054656019e+02
>>> 9 SNES Function norm 3.758975226873e+02
>>> 10 SNES Function norm 3.757032042706e+02
>>> 11 SNES Function norm 3.728798164234e+02
>>> 12 SNES Function norm 3.723078741075e+02
>>> 13 SNES Function norm 3.721848059825e+02
>>> 14 SNES Function norm 3.720227575629e+02
>>> 15 SNES Function norm 3.720051998555e+02
>>> 16 SNES Function norm 3.718945430587e+02
>>> 17 SNES Function norm 3.700412694044e+02
>>> 18 SNES Function norm 3.351964889461e+02
>>> 19 SNES Function norm 3.096016086233e+02
>>> 20 SNES Function norm 3.008410789787e+02
>>> 21 SNES Function norm 2.752316716557e+02
>>> 22 SNES Function norm 2.707658474165e+02
>>> 23 SNES Function norm 2.698436736049e+02
>>> 24 SNES Function norm 2.618233857172e+02
>>> 25 SNES Function norm 2.600121920634e+02
>>> 26 SNES Function norm 2.585046423168e+02
>>> 27 SNES Function norm 2.568551090220e+02
>>> 28 SNES Function norm 2.556404537064e+02
>>> 29 SNES Function norm 2.536353523683e+02
>>> 30 SNES Function norm 2.533596070171e+02
>>> 31 SNES Function norm 2.532324379596e+02
>>> 32 SNES Function norm 2.531842335211e+02
>>> 33 SNES Function norm 2.531684527520e+02
>>> 34 SNES Function norm 2.531637604618e+02
>>> 35 SNES Function norm 2.531624767821e+02
>>> 36 SNES Function norm 2.531621359093e+02
>>> 37 SNES Function norm 2.531620504925e+02
>>> 38 SNES Function norm 2.531620350055e+02
>>> 39 SNES Function norm 2.531620310522e+02
>>> 40 SNES Function norm 2.531620300471e+02
>>> 41 SNES Function norm 2.531620298084e+02
>>> 42 SNES Function norm 2.531620297478e+02
>>> 43 SNES Function norm 2.531620297324e+02
>>> 44 SNES Function norm 2.531620297303e+02
>>> 45 SNES Function norm 2.531620297302e+02
>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations
>>> 45
>>> 0 SNES Function norm 9.636339304380e+03
>>> 1 SNES Function norm 8.997731184634e+03
>>> 2 SNES Function norm 8.120498349232e+03
>>> 3 SNES Function norm 7.322379894820e+03
>>> 4 SNES Function norm 6.599581599149e+03
>>> 5 SNES Function norm 6.374872854688e+03
>>> 6 SNES Function norm 6.372518007653e+03
>>> 7 SNES Function norm 6.073996314301e+03
>>> 8 SNES Function norm 5.635965277054e+03
>>> 9 SNES Function norm 5.155389064046e+03
>>> 10 SNES Function norm 5.080567902638e+03
>>> 11 SNES Function norm 5.058878643969e+03
>>> 12 SNES Function norm 5.058835649793e+03
>>> 13 SNES Function norm 5.058491285707e+03
>>> 14 SNES Function norm 5.057452865337e+03
>>> 15 SNES Function norm 5.057226140688e+03
>>> 16 SNES Function norm 5.056651272898e+03
>>> 17 SNES Function norm 5.056575190057e+03
>>> 18 SNES Function norm 5.056574632598e+03
>>> 19 SNES Function norm 5.056574520229e+03
>>> 20 SNES Function norm 5.056574492569e+03
>>> 21 SNES Function norm 5.056574485124e+03
>>> 22 SNES Function norm 5.056574483029e+03
>>> 23 SNES Function norm 5.056574482427e+03
>>> 24 SNES Function norm 5.056574482302e+03
>>> 25 SNES Function norm 5.056574482287e+03
>>> 26 SNES Function norm 5.056574482282e+03
>>> 27 SNES Function norm 5.056574482281e+03
>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations
>>> 27
>>> SNES Object: 1 MPI processes
>>> type: newtonls
>>> maximum iterations=50, maximum function evaluations=10000
>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>> total number of linear solver iterations=28
>>> total number of function evaluations=323
>>> total number of grid sequence refinements=2
>>> SNESLineSearch Object:   1 MPI processes
>>> type: bt
>>> interpolation: cubic
>>> alpha=1.000000e-04
>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15,
>>> lambda=1.000000e-08
>>> maximum iterations=40
>>> KSP Object:   1 MPI processes
>>> type: gmres
>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>> GMRES: happy breakdown tolerance 1e-30
>>> maximum iterations=10000, initial guess is zero
>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>> left preconditioning
>>> using PRECONDITIONED norm type for convergence test
>>> PC Object:   1 MPI processes
>>> type: lu
>>> LU: out-of-place factorization
>>> tolerance for zero pivot 2.22045e-14
>>> matrix ordering: nd
>>> factor fill ratio given 0, needed 0
>>> Factored matrix follows:
>>>  Mat Object:           1 MPI processes
>>>    type: seqaij
>>>    rows=15991, cols=15991
>>>    package used to perform factorization: mumps
>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>    total number of mallocs used during MatSetValues calls =0
>>>      MUMPS run parameters:
>>>        SYM (matrix type):                   0
>>>        PAR (host participation):            1
>>>        ICNTL(1) (output for error):         6
>>>        ICNTL(2) (output of diagnostic msg): 0
>>>        ICNTL(3) (output for global info):   0
>>>        ICNTL(4) (level of printing):        0
>>>        ICNTL(5) (input mat struct):         0
>>>        ICNTL(6) (matrix prescaling):        7
>>>        ICNTL(7) (sequentia matrix ordering):6
>>>        ICNTL(8) (scalling strategy):        77
>>>        ICNTL(10) (max num of refinements):  0
>>>        ICNTL(11) (error analysis):          0
>>>        ICNTL(12) (efficiency control):                         1
>>>        ICNTL(13) (efficiency control):                         0
>>>        ICNTL(14) (percentage of estimated workspace increase): 20
>>>        ICNTL(18) (input mat struct):                           0
>>>        ICNTL(19) (Shur complement info):                       0
>>>        ICNTL(20) (rhs sparse pattern):                         0
>>>        ICNTL(21) (somumpstion struct):                            0
>>>        ICNTL(22) (in-core/out-of-core facility):               0
>>>        ICNTL(23) (max size of memory can be allocated locally):0
>>>        ICNTL(24) (detection of null pivot rows):               0
>>>        ICNTL(25) (computation of a null space basis):          0
>>>        ICNTL(26) (Schur options for rhs or solution):          0
>>>        ICNTL(27) (experimental parameter):                     -8
>>>        ICNTL(28) (use parallel or sequential ordering):        1
>>>        ICNTL(29) (parallel ordering):                          0
>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0
>>>        ICNTL(31) (factors is discarded in the solve phase):    0
>>>        ICNTL(33) (compute determinant):                        0
>>>        CNTL(1) (relative pivoting threshold):      0.01
>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08
>>>        CNTL(3) (absomumpste pivoting threshold):      0
>>>        CNTL(4) (vamumpse of static pivoting):         -1
>>>        CNTL(5) (fixation for null pivots):         0
>>>        RINFO(1) (local estimated flops for the elimination after
>>> analysis):
>>>          [0] 1.95838e+06
>>>        RINFO(2) (local estimated flops for the assembly after
>>> factorization):
>>>          [0]  143924
>>>        RINFO(3) (local estimated flops for the elimination after
>>> factorization):
>>>          [0]  1.95943e+06
>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for
>>> running numerical factorization):
>>>        [0] 7
>>>        INFO(16) (size of (in MB) MUMPS internal data used during
>>> numerical factorization):
>>>          [0] 7
>>>        INFO(23) (num of pivots eliminated on this processor after
>>> factorization):
>>>          [0] 15991
>>>        RINFOG(1) (global estimated flops for the elimination after
>>> analysis): 1.95838e+06
>>>        RINFOG(2) (global estimated flops for the assembly after
>>> factorization): 143924
>>>        RINFOG(3) (global estimated flops for the elimination after
>>> factorization): 1.95943e+06
>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>        INFOG(3) (estimated real workspace for factors on all processors
>>> after analysis): 255801
>>>        INFOG(4) (estimated integer workspace for factors on all
>>> processors after analysis): 127874
>>>        INFOG(5) (estimated maximum front size in the complete tree): 11
>>>        INFOG(6) (number of nodes in the complete tree): 3996
>>>        INFOG(7) (ordering option effectively use after analysis): 6
>>>        INFOG(8) (structural symmetry in percent of the permuted matrix
>>> after analysis): 86
>>>        INFOG(9) (total real/complex workspace to store the matrix
>>> factors after factorization): 255865
>>>        INFOG(10) (total integer space store the matrix factors after
>>> factorization): 127890
>>>        INFOG(11) (order of largest frontal matrix after factorization):
>>> 11
>>>        INFOG(12) (number of off-diagonal pivots): 19
>>>        INFOG(13) (number of delayed pivots after factorization): 8
>>>        INFOG(14) (number of memory compress after factorization): 0
>>>        INFOG(15) (number of steps of iterative refinement after
>>> solution): 0
>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for
>>> factorization after analysis: value on the most memory consuming
>>> processor): 7
>>>        INFOG(17) (estimated size of all MUMPS internal data for
>>> factorization after analysis: sum over all processors): 7
>>>        INFOG(18) (size of all MUMPS internal data allocated during
>>> factorization: value on the most memory consuming processor): 7
>>>        INFOG(19) (size of all MUMPS internal data allocated during
>>> factorization: sum over all processors): 7
>>>        INFOG(20) (estimated number of entries in the factors): 255801
>>>        INFOG(21) (size in MB of memory effectively used during
>>> factorization - value on the most memory consuming processor): 7
>>>        INFOG(22) (size in MB of memory effectively used during
>>> factorization - sum over all processors): 7
>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0
>>>
>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1
>>>
>>>        INFOG(25) (after factorization: number of pivots modified by
>>> static pivoting): 0
>>>        INFOG(28) (after factorization: number of null pivots
>>> encountered): 0
>>>        INFOG(29) (after factorization: effective number of entries in
>>> the factors (sum over all processors)): 255865
>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used
>>> during solution phase): 5, 5
>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>        INFOG(34) (exponent of the determinant if determinant is
>>> requested): 0
>>> linear system matrix = precond matrix:
>>> Mat Object:     1 MPI processes
>>> type: seqaij
>>> rows=15991, cols=15991
>>> total: nonzeros=223820, allocated nonzeros=431698
>>> total number of mallocs used during MatSetValues calls =15991
>>> using I-node routines: found 4000 nodes, limit used is 5
>>>
>>>
>>>
>>>
>>> -gideon
>>>
>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>> wrote:
>>> Barry,
>>>
>>> I finally got a chance to really try using the grid sequencing within my
>>> code.  I find that, in some cases, even if it can solve successfully on the
>>> coarsest mesh, the SNES fails, usually due to a line search failure, when
>>> it tries to compute along the grid sequence.  Would you have any
>>> suggestions?
>>>
>>> I apologize if I have asked before, but can you give me -snes_view for
>>> the solver? I could not find it in the email thread.
>>>
>>> I would suggest trying to fiddle with the line search, or precondition
>>> it with Richardson. It would be nice to see -snes_monitor
>>> for the runs that fail, and then we can break down the residual into
>>> fields and look at it again (if my custom residual monitor
>>> does not work we can write one easily). Seeing which part of the
>>> residual does not converge is key to designing the NASM
>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai,
>>> present it. We need better monitoring in PETSc.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>> -gideon
>>>
>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com>
>>> wrote:
>>>
>>> Yes, if i continue in this parameter on the coarse mesh, I can generally
>>> solve at all values. I do find that I need to do some amount of
>>> continuation to solve near the endpoint.  The problem is that on the coarse
>>> mesh, things are not fully resolved at all the values along the
>>> continuation parameter, and I would like to do refinement.
>>>
>>> One subtlety is that I actually want the intermediate continuation
>>> solutions  too.  Currently, without doing any grid sequence, I compute
>>> each, write it to disk, and then go on to the next one.  So I now need to
>>> go back an refine them.  I was thinking that perhaps I could refine them on
>>> the fly, dump them to disk, and use the coarse solution as the starting
>>> guess at the next iteration, but that would seem to require resetting the
>>> snes back to the coarse grid.
>>>
>>> The alternative would be to just script the mesh refinement in a post
>>> processing stage, where each value of the continuation is parameter is
>>> loaded on the coarse mesh, and refined.  Perhaps that?s the most practical
>>> thing to do.
>>>
>>>
>>> I would do the following. Create your DM and create a SNES that will do
>>> the continuation
>>>
>>> loop over continuation parameter
>>>
>>> SNESSolve(snes,NULL,Ucoarse);
>>>
>>> if (you decide you want to see the refined solution at this continuation
>>> point) {
>>>    SNESCreate(comm,&snesrefine);
>>>    SNESSetDM()
>>>    etc
>>>    SNESSetGridSequence(snesrefine,)
>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>    SNESGetSolution(snesrefine,&Ufine);
>>>    VecView(Ufine or do whatever you want to do with the Ufine at that
>>> continuation point
>>>    SNESDestroy(snesrefine);
>>> end if
>>>
>>> end loop over continuation parameter.
>>>
>>> Barry
>>>
>>>
>>> -gideon
>>>
>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>>
>>> 3.  This problem is actually part of a continuation problem that roughly
>>> looks like this
>>>
>>> for( continuation parameter p = 0 to 1){
>>>
>>> solve with parameter p_i using solution from p_{i-1},
>>> }
>>>
>>> What I would like to do is to start the solver, for each value of
>>> parameter p_i on the coarse mesh, and then do grid sequencing on that.  But
>>> it appears that after doing grid sequencing on the initial p_0 = 0, the
>>> SNES is set to use the finer mesh.
>>>
>>>
>>> So you are using continuation to give you a good enough initial guess on
>>> the coarse level to even get convergence on the coarse level? First I would
>>> check if you even need the continuation (or can you not even solve the
>>> coarse problem without it).
>>>
>>> If you do need the continuation then you will need to tweak how you do
>>> the grid sequencing. I think this will work:
>>>
>>> Do not use -snes_grid_sequencing
>>>
>>> Run SNESSolve() as many times as you want with your continuation
>>> parameter. This will all happen on the coarse mesh.
>>>
>>> Call SNESSetGridSequence()
>>>
>>> Then call SNESSolve() again and it will do one solve on the coarse level
>>> and then interpolate to the next level etc.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/4680842f/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Sep  8 21:58:58 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 Sep 2015 21:58:58 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
Message-ID: <2C740D3B-AC9B-4BFA-B91F-4F715882B8D8@mcs.anl.gov>


  yeah but the crappy nonlinear convergence could easily be do to a bad Jacobian and 9 times out of 10 bad Newton convergence comes from a bad Jacobian.

> On Sep 8, 2015, at 9:53 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Sep 8, 2015 at 9:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> Your comment about the different pieces scaling different made me think of something, and I made a slight modification.  It still fails, but the conditioning looks much better, no?
> 
> This is great. Now its clear that the nonlinear convergence is crappy.
> 
> Lets start with looking at the distribution of residuals. Take iterate 50,
> which is as converged as you get and save the residual (VecView with
> binary). Then we can load it up and play with it. I recommend
> 
>   a) Norms for both halves
> 
>   b) Plot of the big half
> 
> What usually happens is that some small part is screwing up, but you
> have to iterate the whole system, and finding the descent direction is
> hard. If you select the problem variables, Newton will converge, and then
> the big system will converge once the problem is removed. This is the classic
> "coarse basis selection" problem in the nonlinear regime.
> 
>    Matt
> 
>   Thanks,
> 
>     Matt
>  
> 
>       0 SNES Function norm 4.027474333769e-09 
>         0 KSP preconditioned resid norm 9.390682304142e-09 true resid norm 4.027474333769e-09 ||r(i)||/||b|| 1.000000000000e+00
>         1 KSP preconditioned resid norm 4.937459249116e-20 true resid norm 3.737772204700e-19 ||r(i)||/||b|| 9.280685350023e-11
>       1 SNES Function norm 2.666655509899e-12 
>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>     0 SNES Function norm 5.066222213175e+03 
>       0 KSP preconditioned resid norm 3.135087050015e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.220576053756e-11 true resid norm 1.206309587478e-07 ||r(i)||/||b|| 2.381083056997e-11
>     1 SNES Function norm 8.484697184819e+02 
>       0 KSP preconditioned resid norm 2.668083368240e+02 true resid norm 8.484697184819e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.388166746665e-10 true resid norm 2.735732864000e-08 ||r(i)||/||b|| 3.224314085004e-11
>     2 SNES Function norm 6.549559723821e+02 
>       0 KSP preconditioned resid norm 1.552154273063e+02 true resid norm 6.549559723821e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.740227244709e-10 true resid norm 1.800722435354e-08 ||r(i)||/||b|| 2.749379364852e-11
>     3 SNES Function norm 5.770723278635e+02 
>       0 KSP preconditioned resid norm 1.348596653013e+02 true resid norm 5.770723278635e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.142365551700e-10 true resid norm 7.387664544933e-09 ||r(i)||/||b|| 1.280197331985e-11
>     4 SNES Function norm 5.237702241066e+02 
>       0 KSP preconditioned resid norm 1.524401223470e+02 true resid norm 5.237702241066e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.168592748079e-10 true resid norm 9.512653260875e-09 ||r(i)||/||b|| 1.816188248788e-11
>     5 SNES Function norm 4.753909020302e+02 
>       0 KSP preconditioned resid norm 1.864261890160e+02 true resid norm 4.753909020302e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.744912816503e-10 true resid norm 1.889384889937e-08 ||r(i)||/||b|| 3.974381675939e-11
>     6 SNES Function norm 4.221784591168e+02 
>       0 KSP preconditioned resid norm 2.550992140633e+02 true resid norm 4.221784591168e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.697718851690e-10 true resid norm 1.666255394677e-08 ||r(i)||/||b|| 3.946803439861e-11
>     7 SNES Function norm 3.806525080909e+02 
>       0 KSP preconditioned resid norm 5.183605288372e+02 true resid norm 3.806525080909e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.240268561213e-09 true resid norm 1.041310587019e-07 ||r(i)||/||b|| 2.735593657958e-10
>     8 SNES Function norm 3.762054656363e+02 
>       0 KSP preconditioned resid norm 5.320776249886e+03 true resid norm 3.762054656363e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.328856456273e-09 true resid norm 7.860313076537e-08 ||r(i)||/||b|| 2.089367059897e-10
>     9 SNES Function norm 3.758975227207e+02 
>       0 KSP preconditioned resid norm 6.252711400154e+03 true resid norm 3.758975227207e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.900963654627e-09 true resid norm 1.347299589292e-07 ||r(i)||/||b|| 3.584220453330e-10
>    10 SNES Function norm 3.757032042008e+02 
>       0 KSP preconditioned resid norm 5.890497230519e+02 true resid norm 3.757032042008e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.497822807230e-09 true resid norm 3.605311757144e-08 ||r(i)||/||b|| 9.596169840535e-11
>    11 SNES Function norm 3.728798164183e+02 
>       0 KSP preconditioned resid norm 1.081356468099e+03 true resid norm 3.728798164183e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.713971332514e-09 true resid norm 3.705072372867e-08 ||r(i)||/||b|| 9.936371478768e-11
>    12 SNES Function norm 3.723078736486e+02 
>       0 KSP preconditioned resid norm 3.745123875883e+03 true resid norm 3.723078736486e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.885886422519e-08 true resid norm 2.837206039485e-07 ||r(i)||/||b|| 7.620591022374e-10
>    13 SNES Function norm 3.721848051981e+02 
>       0 KSP preconditioned resid norm 3.495480420016e+03 true resid norm 3.721848051981e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.874759607493e-08 true resid norm 2.751735241072e-07 ||r(i)||/||b|| 7.393464759014e-10
>    14 SNES Function norm 3.720227571463e+02 
>       0 KSP preconditioned resid norm 1.265998097646e+04 true resid norm 3.720227571463e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.125038941589e-08 true resid norm 5.141611324046e-07 ||r(i)||/||b|| 1.382069033488e-09
>    15 SNES Function norm 3.720051993666e+02 
>       0 KSP preconditioned resid norm 2.912369751490e+03 true resid norm 3.720051993666e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.170907138699e-09 true resid norm 3.623872061163e-08 ||r(i)||/||b|| 9.741455407971e-11
>    16 SNES Function norm 3.718945615434e+02 
>       0 KSP preconditioned resid norm 1.082053028195e+03 true resid norm 3.718945615434e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.660982704773e-09 true resid norm 9.903133228018e-08 ||r(i)||/||b|| 2.662887348210e-10
>    17 SNES Function norm 3.700412140239e+02 
>       0 KSP preconditioned resid norm 3.676306553630e+01 true resid norm 3.700412140239e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.389666775474e-10 true resid norm 4.384613797840e-10 ||r(i)||/||b|| 1.184898771183e-12
>    18 SNES Function norm 3.351961239413e+02 
>       0 KSP preconditioned resid norm 1.071519127966e+02 true resid norm 3.351961239413e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.391049839274e-11 true resid norm 1.771280721778e-09 ||r(i)||/||b|| 5.284311468017e-12
>    19 SNES Function norm 3.096040290055e+02 
>       0 KSP preconditioned resid norm 3.362840348158e+02 true resid norm 3.096040290055e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.847034649159e-10 true resid norm 3.697317355249e-09 ||r(i)||/||b|| 1.194208411023e-11
>    20 SNES Function norm 3.008687462984e+02 
>       0 KSP preconditioned resid norm 7.161404453567e+01 true resid norm 3.008687462984e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.352839847013e-11 true resid norm 7.837384020140e-10 ||r(i)||/||b|| 2.604917963917e-12
>    21 SNES Function norm 2.752608806757e+02 
>       0 KSP preconditioned resid norm 1.727563200247e+02 true resid norm 2.752608806757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.663026698374e-10 true resid norm 5.807801532466e-09 ||r(i)||/||b|| 2.109926233692e-11
>    22 SNES Function norm 2.707559177016e+02 
>       0 KSP preconditioned resid norm 4.125046548626e+02 true resid norm 2.707559177016e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.109007703208e-10 true resid norm 5.875389803064e-09 ||r(i)||/||b|| 2.169994972940e-11
>    23 SNES Function norm 2.698855074973e+02 
>       0 KSP preconditioned resid norm 7.414116220947e+01 true resid norm 2.698855074973e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.222534270039e-11 true resid norm 6.928234778488e-10 ||r(i)||/||b|| 2.567101450810e-12
>    24 SNES Function norm 2.616892369202e+02 
>       0 KSP preconditioned resid norm 1.819161466540e+02 true resid norm 2.616892369202e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.013662651542e-09 true resid norm 4.195842561304e-09 ||r(i)||/||b|| 1.603368411588e-11
>    25 SNES Function norm 2.600248325147e+02 
>       0 KSP preconditioned resid norm 1.265123744997e+02 true resid norm 2.600248325147e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.572744639399e-09 true resid norm 7.386966838391e-09 ||r(i)||/||b|| 2.840869761150e-11
>    26 SNES Function norm 2.593065715720e+02 
>       0 KSP preconditioned resid norm 1.314965938655e+02 true resid norm 2.593065715720e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.156889834113e-10 true resid norm 3.157778817609e-09 ||r(i)||/||b|| 1.217778168314e-11
>    27 SNES Function norm 2.517348459439e+02 
>       0 KSP preconditioned resid norm 2.618005944700e+01 true resid norm 2.517348459439e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.483924363849e-11 true resid norm 7.418968110702e-10 ||r(i)||/||b|| 2.947135936976e-12
>    28 SNES Function norm 2.298635352641e+02 
>       0 KSP preconditioned resid norm 8.770042104744e+01 true resid norm 2.298635352641e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.098440763657e-09 true resid norm 3.936819814733e-09 ||r(i)||/||b|| 1.712676962968e-11
>    29 SNES Function norm 2.269871118080e+02 
>       0 KSP preconditioned resid norm 2.285655807501e+02 true resid norm 2.269871118080e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.216116049541e-08 true resid norm 9.327420966977e-08 ||r(i)||/||b|| 4.109229327024e-10
>    30 SNES Function norm 2.252105956612e+02 
>       0 KSP preconditioned resid norm 8.597156834231e+01 true resid norm 2.252105956612e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.431615405953e-09 true resid norm 4.297181487549e-09 ||r(i)||/||b|| 1.908072519826e-11
>    31 SNES Function norm 2.251883212710e+02 
>       0 KSP preconditioned resid norm 7.066983917118e+01 true resid norm 2.251883212710e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.998573955346e-10 true resid norm 3.454505271056e-09 ||r(i)||/||b|| 1.534051700176e-11
>    32 SNES Function norm 2.117895362999e+02 
>       0 KSP preconditioned resid norm 3.830985324064e+02 true resid norm 2.117895362999e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.016573825110e-09 true resid norm 1.942657883557e-08 ||r(i)||/||b|| 9.172586698552e-11
>    33 SNES Function norm 2.116404150058e+02 
>       0 KSP preconditioned resid norm 5.329232125733e+02 true resid norm 2.116404150058e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.609537995048e-09 true resid norm 1.451505196923e-08 ||r(i)||/||b|| 6.858355465253e-11
>    34 SNES Function norm 2.115566314991e+02 
>       0 KSP preconditioned resid norm 6.609724236205e+02 true resid norm 2.115566314991e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.254105497370e-10 true resid norm 1.178528765891e-08 ||r(i)||/||b|| 5.570748397436e-11
>    35 SNES Function norm 2.115028944726e+02 
>       0 KSP preconditioned resid norm 8.217040556045e+02 true resid norm 2.115028944726e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.803324342385e-09 true resid norm 2.957366914143e-08 ||r(i)||/||b|| 1.398263093050e-10
>    36 SNES Function norm 2.114773510911e+02 
>       0 KSP preconditioned resid norm 1.091957913212e+03 true resid norm 2.114773510911e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.264170340401e-08 true resid norm 1.635833495717e-07 ||r(i)||/||b|| 7.735265678696e-10
>    37 SNES Function norm 2.114573155596e+02 
>       0 KSP preconditioned resid norm 1.295967660499e+03 true resid norm 2.114573155596e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.787403648651e-08 true resid norm 1.913077973453e-07 ||r(i)||/||b|| 9.047111793648e-10
>    38 SNES Function norm 2.114434864633e+02 
>       0 KSP preconditioned resid norm 1.567323979910e+03 true resid norm 2.114434864633e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.869044911030e-09 true resid norm 3.425540409553e-08 ||r(i)||/||b|| 1.620073744928e-10
>    39 SNES Function norm 2.114337906450e+02 
>       0 KSP preconditioned resid norm 1.835099219771e+03 true resid norm 2.114337906450e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.224665285731e-10 true resid norm 1.690151657152e-08 ||r(i)||/||b|| 7.993763210677e-11
>    40 SNES Function norm 2.114267348757e+02 
>       0 KSP preconditioned resid norm 2.146249211809e+03 true resid norm 2.114267348757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.809299780773e-08 true resid norm 1.157600089098e-07 ||r(i)||/||b|| 5.475183116166e-10
>    41 SNES Function norm 2.114215910347e+02 
>       0 KSP preconditioned resid norm 2.510692990698e+03 true resid norm 2.114215910347e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.485943641401e-08 true resid norm 9.439839396991e-08 ||r(i)||/||b|| 4.464936315535e-10
>    42 SNES Function norm 2.114206676373e+02 
>       0 KSP preconditioned resid norm 3.401544811763e+03 true resid norm 2.114206676373e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.910138749741e-09 true resid norm 3.806926432018e-08 ||r(i)||/||b|| 1.800640625423e-10
>    43 SNES Function norm 2.114186309697e+02 
>       0 KSP preconditioned resid norm 3.955962524189e+03 true resid norm 2.114186309697e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.500201400572e-08 true resid norm 1.570197988244e-07 ||r(i)||/||b|| 7.426961290224e-10
>    44 SNES Function norm 2.114171338084e+02 
>       0 KSP preconditioned resid norm 4.636713819556e+03 true resid norm 2.114171338084e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.248004040828e-09 true resid norm 7.296845022145e-08 ||r(i)||/||b|| 3.451397193171e-10
>    45 SNES Function norm 2.114170015592e+02 
>       0 KSP preconditioned resid norm 6.321621979244e+03 true resid norm 2.114170015592e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.674310076597e-08 true resid norm 1.788726595879e-07 ||r(i)||/||b|| 8.460656345928e-10
>    46 SNES Function norm 2.114164190412e+02 
>       0 KSP preconditioned resid norm 7.414534654352e+03 true resid norm 2.114164190412e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.520165386851e-08 true resid norm 4.326674448851e-07 ||r(i)||/||b|| 2.046517705897e-09
>    47 SNES Function norm 2.114162562951e+02 
>       0 KSP preconditioned resid norm 9.810108530473e+03 true resid norm 2.114162562951e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.690981568660e-08 true resid norm 1.826808079437e-07 ||r(i)||/||b|| 8.640811787375e-10
>    48 SNES Function norm 2.114160137794e+02 
>       0 KSP preconditioned resid norm 1.136099973228e+04 true resid norm 2.114160137794e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.015775310105e-08 true resid norm 1.261980904678e-07 ||r(i)||/||b|| 5.969183138583e-10
>    49 SNES Function norm 2.114158331092e+02 
>       0 KSP preconditioned resid norm 1.315560026827e+04 true resid norm 2.114158331092e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.131165057283e-08 true resid norm 3.097361970470e-07 ||r(i)||/||b|| 1.465056767470e-09
>    50 SNES Function norm 2.114158223286e+02 
>       0 KSP preconditioned resid norm 1.794717411653e+04 true resid norm 2.114158223286e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.070352092082e-07 true resid norm 6.224398071454e-07 ||r(i)||/||b|| 2.944149592446e-09
>    51 SNES Function norm 2.114157500757e+02 
>       0 KSP preconditioned resid norm 2.077631649185e+04 true resid norm 2.114157500757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.358541701826e-08 true resid norm 5.604873460299e-07 ||r(i)||/||b|| 2.651114431301e-09
>    52 SNES Function norm 2.114156961871e+02 
>       0 KSP preconditioned resid norm 2.405048247398e+04 true resid norm 2.114156961871e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.439555573169e-08 true resid norm 3.896791261159e-07 ||r(i)||/||b|| 1.843189191455e-09
>    53 SNES Function norm 2.114156592651e+02 
>       0 KSP preconditioned resid norm 2.914886710376e+04 true resid norm 2.114156592651e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.436687722935e-07 true resid norm 1.370484810325e-06 ||r(i)||/||b|| 6.482418639609e-09
>    54 SNES Function norm 2.114156319126e+02 
>       0 KSP preconditioned resid norm 3.373835363786e+04 true resid norm 2.114156319126e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.028533154395e-07 true resid norm 6.668879459515e-07 ||r(i)||/||b|| 3.154392794508e-09
>    55 SNES Function norm 2.114156115019e+02 
>       0 KSP preconditioned resid norm 3.904963797200e+04 true resid norm 2.114156115019e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.986152192528e-08 true resid norm 4.508900491313e-07 ||r(i)||/||b|| 2.132718799374e-09
>    56 SNES Function norm 2.114155964342e+02 
>       0 KSP preconditioned resid norm 4.594596937760e+04 true resid norm 2.114155964342e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.567731897537e-08 true resid norm 5.054454305445e-07 ||r(i)||/||b|| 2.390766996709e-09
>    57 SNES Function norm 2.114155854345e+02 
>       0 KSP preconditioned resid norm 5.317614782539e+04 true resid norm 2.114155854345e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.460625544912e-07 true resid norm 9.171801951021e-07 ||r(i)||/||b|| 4.338280894556e-09
>    58 SNES Function norm 2.114155777135e+02 
>       0 KSP preconditioned resid norm 6.400017583847e+04 true resid norm 2.114155777135e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.093685259962e-08 true resid norm 6.506435082837e-07 ||r(i)||/||b|| 3.077557081273e-09
>    59 SNES Function norm 2.114155720468e+02 
>       0 KSP preconditioned resid norm 7.406807436961e+04 true resid norm 2.114155720468e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.571128588743e-07 true resid norm 1.516640238741e-06 ||r(i)||/||b|| 7.173739493538e-09
>    60 SNES Function norm 2.114155680388e+02 
>       0 KSP preconditioned resid norm 8.892466432276e+04 true resid norm 2.114155680388e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.335222631846e-08 true resid norm 8.161361513986e-07 ||r(i)||/||b|| 3.860340839464e-09
>    61 SNES Function norm 2.114155651044e+02 
>       0 KSP preconditioned resid norm 1.029106627628e+05 true resid norm 2.114155651044e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.244108037270e-07 true resid norm 3.009616926997e-06 ||r(i)||/||b|| 1.423555037450e-08
>    62 SNES Function norm 2.114155629136e+02 
>       0 KSP preconditioned resid norm 1.190953464375e+05 true resid norm 2.114155629136e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.108162336627e-08 true resid norm 9.193387836453e-07 ||r(i)||/||b|| 4.348491525295e-09
>    63 SNES Function norm 2.114155612779e+02 
>       0 KSP preconditioned resid norm 1.378243866677e+05 true resid norm 2.114155612779e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.138603793608e-07 true resid norm 2.482322226118e-06 ||r(i)||/||b|| 1.174143573497e-08
>    64 SNES Function norm 2.114155602404e+02 
>       0 KSP preconditioned resid norm 1.698559088929e+05 true resid norm 2.114155602404e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.608494949489e-08 true resid norm 1.125534374356e-06 ||r(i)||/||b|| 5.323801015765e-09
>    65 SNES Function norm 2.114155594364e+02 
>       0 KSP preconditioned resid norm 1.965648734846e+05 true resid norm 2.114155594364e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.689853079655e-07 true resid norm 2.539748843840e-06 ||r(i)||/||b|| 1.201306493529e-08
>    66 SNES Function norm 2.114155588361e+02 
>       0 KSP preconditioned resid norm 2.274726625349e+05 true resid norm 2.114155588361e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.937782777025e-07 true resid norm 5.126870538363e-06 ||r(i)||/||b|| 2.425020450996e-08
>    67 SNES Function norm 2.114155583879e+02 
>       0 KSP preconditioned resid norm 2.632394448333e+05 true resid norm 2.114155583879e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.229084794159e-07 true resid norm 3.026195587359e-06 ||r(i)||/||b|| 1.431396823599e-08
>    68 SNES Function norm 2.114155581765e+02 
>       0 KSP preconditioned resid norm 3.367643572924e+05 true resid norm 2.114155581765e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.632794900620e-08 true resid norm 2.446819364813e-06 ||r(i)||/||b|| 1.157350663271e-08
>    69 SNES Function norm 2.114155579720e+02 
>       0 KSP preconditioned resid norm 3.897127303190e+05 true resid norm 2.114155579720e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.933486330809e-06 true resid norm 2.730183810468e-05 ||r(i)||/||b|| 1.291382638372e-07
>    70 SNES Function norm 2.114155578193e+02 
>       0 KSP preconditioned resid norm 4.509848215072e+05 true resid norm 2.114155578193e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.706389710731e-08 true resid norm 3.785217910694e-06 ||r(i)||/||b|| 1.790415970205e-08
>    71 SNES Function norm 2.114155577054e+02 
>       0 KSP preconditioned resid norm 5.242625554614e+05 true resid norm 2.114155577054e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.321749665453e-06 true resid norm 9.069981074375e-06 ||r(i)||/||b|| 4.290119976419e-08
>    72 SNES Function norm 2.114155576210e+02 
>       0 KSP preconditioned resid norm 6.066871925199e+05 true resid norm 2.114155576210e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.327204488687e-07 true resid norm 7.090997425424e-06 ||r(i)||/||b|| 3.354056581842e-08
>    73 SNES Function norm 2.114155575580e+02 
>       0 KSP preconditioned resid norm 7.020695777561e+05 true resid norm 2.114155575580e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.123816847857e-06 true resid norm 8.125899637337e-06 ||r(i)||/||b|| 3.843567489166e-08
>    74 SNES Function norm 2.114155575109e+02 
>       0 KSP preconditioned resid norm 8.124474305522e+05 true resid norm 2.114155575109e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.293550435673e-07 true resid norm 7.844829319667e-06 ||r(i)||/||b|| 3.710620643072e-08
>    75 SNES Function norm 2.114155574865e+02 
>       0 KSP preconditioned resid norm 1.029314742970e+06 true resid norm 2.114155574865e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.033392384072e-07 true resid norm 9.399925240813e-06 ||r(i)||/||b|| 4.446184260312e-08
>    76 SNES Function norm 2.114155574646e+02 
>       0 KSP preconditioned resid norm 1.191137352032e+06 true resid norm 2.114155574646e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.636526184597e-07 true resid norm 1.077765913224e-05 ||r(i)||/||b|| 5.097855267366e-08
>    77 SNES Function norm 2.114155574575e+02 
>       0 KSP preconditioned resid norm 1.563965951801e+06 true resid norm 2.114155574575e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.137216998536e-06 true resid norm 5.108848518118e-05 ||r(i)||/||b|| 2.416496013612e-07
>    78 SNES Function norm 2.114155574558e+02 
>       0 KSP preconditioned resid norm 2.108895892585e+06 true resid norm 2.114155574558e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.138935971807e-05 true resid norm 6.467637231732e-05 ||r(i)||/||b|| 3.059205911601e-07
>    79 SNES Function norm 2.114155574511e+02 
>       0 KSP preconditioned resid norm 2.564542138115e+06 true resid norm 2.114155574511e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.391129540980e-06 true resid norm 4.283623913226e-05 ||r(i)||/||b|| 2.026163052933e-07
>    80 SNES Function norm 2.114155574475e+02 
>       0 KSP preconditioned resid norm 2.967732587697e+06 true resid norm 2.114155574475e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.446508619620e-06 true resid norm 5.164837794317e-05 ||r(i)||/||b|| 2.442979058246e-07
>    81 SNES Function norm 2.114155574449e+02 
>       0 KSP preconditioned resid norm 3.434290246612e+06 true resid norm 2.114155574449e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.781979196562e-06 true resid norm 4.886329177979e-05 ||r(i)||/||b|| 2.311243901364e-07
>    82 SNES Function norm 2.114155574429e+02 
>       0 KSP preconditioned resid norm 3.974205574181e+06 true resid norm 2.114155574429e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.425678262963e-05 true resid norm 8.517034218173e-05 ||r(i)||/||b|| 4.028574964485e-07
>    83 SNES Function norm 2.114155574416e+02 
>       0 KSP preconditioned resid norm 4.785116089506e+06 true resid norm 2.114155574416e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.130667094145e-05 true resid norm 1.264608296603e-04 ||r(i)||/||b|| 5.981623641639e-07
>    84 SNES Function norm 2.114155574406e+02 
>       0 KSP preconditioned resid norm 5.537350742612e+06 true resid norm 2.114155574406e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.299149833543e-06 true resid norm 5.419308321331e-05 ||r(i)||/||b|| 2.563344148812e-07
>    85 SNES Function norm 2.114155574399e+02 
>       0 KSP preconditioned resid norm 6.767161611151e+06 true resid norm 2.114155574399e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.781986133631e-05 true resid norm 1.232616969009e-04 ||r(i)||/||b|| 5.830303994346e-07
>    86 SNES Function norm 2.114155574394e+02 
>       0 KSP preconditioned resid norm 7.867884017146e+06 true resid norm 2.114155574394e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.002696664504e-05 true resid norm 1.398533205346e-04 ||r(i)||/||b|| 6.615091255747e-07
>    87 SNES Function norm 2.114155574390e+02 
>       0 KSP preconditioned resid norm 9.102289060164e+06 true resid norm 2.114155574390e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.231201322143e-05 true resid norm 1.368407286352e-04 ||r(i)||/||b|| 6.472595030036e-07
>    88 SNES Function norm 2.114155574387e+02 
>       0 KSP preconditioned resid norm 1.063720954841e+07 true resid norm 2.114155574387e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.517853677131e-05 true resid norm 1.273061147947e-04 ||r(i)||/||b|| 6.021605805032e-07
>    89 SNES Function norm 2.114155574385e+02 
>       0 KSP preconditioned resid norm 1.230866787650e+07 true resid norm 2.114155574385e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.745229646688e-05 true resid norm 1.389443348166e-04 ||r(i)||/||b|| 6.572096041560e-07
>    90 SNES Function norm 2.114155574384e+02 
>       0 KSP preconditioned resid norm 1.436772950622e+07 true resid norm 2.114155574384e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.955301916175e-05 true resid norm 5.633754040143e-04 ||r(i)||/||b|| 2.664777421494e-06
>    91 SNES Function norm 2.114155574383e+02 
>       0 KSP preconditioned resid norm 1.662010558966e+07 true resid norm 2.114155574383e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.147116976734e-05 true resid norm 1.351152564949e-04 ||r(i)||/||b|| 6.390979837628e-07
>    92 SNES Function norm 2.114155574382e+02 
>       0 KSP preconditioned resid norm 1.923255924422e+07 true resid norm 2.114155574382e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.030030646485e-05 true resid norm 1.877965090103e-04 ||r(i)||/||b|| 8.882814078868e-07
>    93 SNES Function norm 2.114155574381e+02 
>       0 KSP preconditioned resid norm 2.226793244995e+07 true resid norm 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.411985876631e-04 true resid norm 1.342893408929e-03 ||r(i)||/||b|| 6.351913857247e-06
>    94 SNES Function norm 2.114155574381e+02 
>       0 KSP preconditioned resid norm 2.576953439486e+07 true resid norm 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.147595096922e-05 true resid norm 4.578229507611e-04 ||r(i)||/||b|| 2.165512114193e-06
>    95 SNES Function norm 2.114155574380e+02 
>       0 KSP preconditioned resid norm 2.981662207354e+07 true resid norm 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.456246888386e-06 true resid norm 2.856742906943e-04 ||r(i)||/||b|| 1.351245358460e-06
>    96 SNES Function norm 2.114155574380e+02 
>       0 KSP preconditioned resid norm 3.451395888978e+07 true resid norm 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.972976681165e-05 true resid norm 3.126288732572e-04 ||r(i)||/||b|| 1.478741096662e-06
>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 96
> 
> -gideon
> 
>> On Sep 8, 2015, at 10:31 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Tue, Sep 8, 2015 at 9:19 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> Ok, going back and running the problem with the flags, -snes_fd -snes_mf_operator -ksp_monitor_true_residual -snes_max_funcs 500000, during the grid sequence, I still get:
>> 
>> Lets try using -pc_type lu instead of MUMPS, just to make sure. If we still
>> get this, then the problem is really horribly conditioned without being singular.
>> We are losing all precision for some reason. Is it possible that the two halves
>> are so differently scaled that we make no progress on one with LU?
>> 
>>   Thanks,
>> 
>>      Matt
>>  
>>       0 SNES Function norm 9.131726454160e-09 
>>         0 KSP preconditioned resid norm 4.201012306235e-09 true resid norm 9.131726454160e-09 ||r(i)||/||b|| 1.000000000000e+00
>>         1 KSP preconditioned resid norm 4.811278613811e-14 true resid norm 7.117495013042e-13 ||r(i)||/||b|| 7.794249037978e-05
>>         2 KSP preconditioned resid norm 1.729322442321e-19 true resid norm 2.611605096203e-13 ||r(i)||/||b|| 2.859924800982e-05
>>       1 SNES Function norm 2.066497600915e-12 
>>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>     0 SNES Function norm 5.066222213176e+03 
>>       0 KSP preconditioned resid norm 3.135051257976e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.425321207844e-04 true resid norm 1.654961474093e+00 ||r(i)||/||b|| 3.266657885218e-04
>>       2 KSP preconditioned resid norm 2.739130995469e-09 true resid norm 7.943827483388e-03 ||r(i)||/||b|| 1.567998234015e-06
>>     1 SNES Function norm 8.487402654330e+02 
>>       0 KSP preconditioned resid norm 2.670376525819e+02 true resid norm 8.487402654330e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.518628784729e-02 true resid norm 1.820466086409e+00 ||r(i)||/||b|| 2.144903641964e-03
>>       2 KSP preconditioned resid norm 1.680654269556e-07 true resid norm 1.635299027240e-01 ||r(i)||/||b|| 1.926736710678e-04
>>     2 SNES Function norm 6.547660268557e+02 
>>       0 KSP preconditioned resid norm 1.529984375428e+02 true resid norm 6.547660268557e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.106652647444e-02 true resid norm 9.350438798847e-01 ||r(i)||/||b|| 1.428058026124e-03
>>       2 KSP preconditioned resid norm 1.739404692247e-07 true resid norm 6.642022841611e-02 ||r(i)||/||b|| 1.014411647701e-04
>>     3 SNES Function norm 5.771035131885e+02 
>>       0 KSP preconditioned resid norm 1.360697738776e+02 true resid norm 5.771035131885e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.323138791370e-03 true resid norm 6.419077863137e-02 ||r(i)||/||b|| 1.112292286642e-04
>>     4 SNES Function norm 5.238679723156e+02 
>>       0 KSP preconditioned resid norm 1.528156345475e+02 true resid norm 5.238679723156e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.115694692015e-02 true resid norm 1.230231531838e+00 ||r(i)||/||b|| 2.348361795054e-03
>>       2 KSP preconditioned resid norm 5.444815701049e-08 true resid norm 6.640979326953e-02 ||r(i)||/||b|| 1.267681873660e-04
>>     5 SNES Function norm 4.755828064476e+02 
>>       0 KSP preconditioned resid norm 1.872910183528e+02 true resid norm 4.755828064476e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.182180755863e-02 true resid norm 5.918459188490e-01 ||r(i)||/||b|| 1.244464498769e-03
>>       2 KSP preconditioned resid norm 7.426717097996e-08 true resid norm 8.473303784961e-02 ||r(i)||/||b|| 1.781667392111e-04
>>     6 SNES Function norm 4.224257767287e+02 
>>       0 KSP preconditioned resid norm 2.578229460575e+02 true resid norm 4.224257767287e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.258348180527e-02 true resid norm 3.257118054248e-01 ||r(i)||/||b|| 7.710509712432e-04
>>       2 KSP preconditioned resid norm 1.122972268815e-07 true resid norm 1.274729638407e-01 ||r(i)||/||b|| 3.017641698568e-04
>>     7 SNES Function norm 3.811223987907e+02 
>>       0 KSP preconditioned resid norm 5.218684581634e+02 true resid norm 3.811223987907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.089005298384e-01 true resid norm 5.194667956652e+00 ||r(i)||/||b|| 1.362992039601e-02
>>       2 KSP preconditioned resid norm 7.570453480881e-07 true resid norm 2.863356359719e-01 ||r(i)||/||b|| 7.512957435207e-04
>>     8 SNES Function norm 3.771290745624e+02 
>>       0 KSP preconditioned resid norm 5.827455360446e+03 true resid norm 3.771290745624e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.620225652579e-01 true resid norm 1.332007773441e+01 ||r(i)||/||b|| 3.531967867995e-02
>>       2 KSP preconditioned resid norm 3.596801636560e-06 true resid norm 3.425465452374e+00 ||r(i)||/||b|| 9.083005483862e-03
>>     9 SNES Function norm 3.767897776540e+02 
>>       0 KSP preconditioned resid norm 5.185115189077e+03 true resid norm 3.767897776540e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.021105455600e-01 true resid norm 1.043813038739e+01 ||r(i)||/||b|| 2.770279611187e-02
>>       2 KSP preconditioned resid norm 6.951879108628e-06 true resid norm 2.816562934830e+00 ||r(i)||/||b|| 7.475157506572e-03
>>    10 SNES Function norm 3.746968158051e+02 
>>       0 KSP preconditioned resid norm 8.165048723911e+02 true resid norm 3.746968158051e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.177528145230e-01 true resid norm 2.394044923481e+00 ||r(i)||/||b|| 6.389285476945e-03
>>       2 KSP preconditioned resid norm 7.325770200602e-07 true resid norm 4.877299135281e-01 ||r(i)||/||b|| 1.301665487816e-03
>>    11 SNES Function norm 3.726877791108e+02 
>>       0 KSP preconditioned resid norm 5.024547551936e+02 true resid norm 3.726877791108e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.640017871568e-01 true resid norm 1.648101091752e+00 ||r(i)||/||b|| 4.422203206351e-03
>>       2 KSP preconditioned resid norm 1.005738633834e-06 true resid norm 2.935508517682e-01 ||r(i)||/||b|| 7.876589151074e-04
>>    12 SNES Function norm 3.668832918639e+02 
>>       0 KSP preconditioned resid norm 7.762104953873e+02 true resid norm 3.668832918639e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.912259329681e-02 true resid norm 1.334600681831e+00 ||r(i)||/||b|| 3.637670919956e-03
>>       2 KSP preconditioned resid norm 7.640320635302e-07 true resid norm 4.471334143387e-01 ||r(i)||/||b|| 1.218734742776e-03
>>    13 SNES Function norm 3.631997889667e+02 
>>       0 KSP preconditioned resid norm 2.515761606628e+02 true resid norm 3.631997889667e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.648297544435e-01 true resid norm 2.866637955219e+00 ||r(i)||/||b|| 7.892730233611e-03
>>       2 KSP preconditioned resid norm 7.305731711999e-07 true resid norm 1.472589913546e-01 ||r(i)||/||b|| 4.054490003243e-04
>>    14 SNES Function norm 3.557844076846e+02 
>>       0 KSP preconditioned resid norm 6.995145168850e+02 true resid norm 3.557844076846e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.626527197886e-02 true resid norm 5.642316830909e-01 ||r(i)||/||b|| 1.585880861848e-03
>>       2 KSP preconditioned resid norm 1.040702263139e-06 true resid norm 4.073703976052e-01 ||r(i)||/||b|| 1.144992272866e-03
>>    15 SNES Function norm 3.534147071188e+02 
>>       0 KSP preconditioned resid norm 2.698507093385e+02 true resid norm 3.534147071188e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.592052963868e-01 true resid norm 1.332346399928e+00 ||r(i)||/||b|| 3.769923472595e-03
>>       2 KSP preconditioned resid norm 5.021635239169e-07 true resid norm 1.573071613558e-01 ||r(i)||/||b|| 4.451064378114e-04
>>    16 SNES Function norm 3.499900270010e+02 
>>       0 KSP preconditioned resid norm 7.295084253442e+01 true resid norm 3.499900270010e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.040248007780e-02 true resid norm 1.427506717563e-01 ||r(i)||/||b|| 4.078706841436e-04
>>       2 KSP preconditioned resid norm 1.246491224109e-07 true resid norm 4.074099442269e-02 ||r(i)||/||b|| 1.164061581177e-04
>>    17 SNES Function norm 3.473017914657e+02 
>>       0 KSP preconditioned resid norm 4.169372990718e+02 true resid norm 3.473017914657e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.140552389441e-01 true resid norm 1.893314948416e+00 ||r(i)||/||b|| 5.451497789361e-03
>>       2 KSP preconditioned resid norm 7.751532205539e-07 true resid norm 2.494992549211e-01 ||r(i)||/||b|| 7.183932275966e-04
>>    18 SNES Function norm 3.464416771335e+02 
>>       0 KSP preconditioned resid norm 2.968698304201e+02 true resid norm 3.464416771335e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.683113115222e-02 true resid norm 2.501212385243e-01 ||r(i)||/||b|| 7.219721385540e-04
>>       2 KSP preconditioned resid norm 5.274081741342e-07 true resid norm 1.712806350336e-01 ||r(i)||/||b|| 4.943996243489e-04
>>    19 SNES Function norm 3.460747441983e+02 
>>       0 KSP preconditioned resid norm 3.707198448214e+02 true resid norm 3.460747441983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.740664131652e-02 true resid norm 9.477346570578e-01 ||r(i)||/||b|| 2.738525919461e-03
>>       2 KSP preconditioned resid norm 4.649501179917e-07 true resid norm 2.217443031583e-01 ||r(i)||/||b|| 6.407410736429e-04
>>    20 SNES Function norm 3.455596398964e+02 
>>       0 KSP preconditioned resid norm 5.000331867417e+02 true resid norm 3.455596398964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.821124737573e-02 true resid norm 7.651194558377e-01 ||r(i)||/||b|| 2.214145888296e-03
>>       2 KSP preconditioned resid norm 5.030283955029e-07 true resid norm 2.918161448112e-01 ||r(i)||/||b|| 8.444740389784e-04
>>    21 SNES Function norm 3.453222949611e+02 
>>       0 KSP preconditioned resid norm 7.399233760260e+02 true resid norm 3.453222949611e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.030913689264e-01 true resid norm 8.930798988128e-01 ||r(i)||/||b|| 2.586221370136e-03
>>       2 KSP preconditioned resid norm 1.497261432173e-06 true resid norm 4.385255355661e-01 ||r(i)||/||b|| 1.269902181136e-03
>>    22 SNES Function norm 3.452850731057e+02 
>>       0 KSP preconditioned resid norm 1.850020325701e+03 true resid norm 3.452850731057e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.670597549481e-01 true resid norm 6.205758169419e+00 ||r(i)||/||b|| 1.797285389027e-02
>>       2 KSP preconditioned resid norm 4.122440586032e-06 true resid norm 1.109676174925e+00 ||r(i)||/||b|| 3.213797124053e-03
>>    23 SNES Function norm 3.452722963704e+02 
>>       0 KSP preconditioned resid norm 3.362446711623e+03 true resid norm 3.452722963704e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.181412468129e+00 true resid norm 1.564006484993e+01 ||r(i)||/||b|| 4.529776936738e-02
>>       2 KSP preconditioned resid norm 1.165193864469e-05 true resid norm 1.869411988347e+00 ||r(i)||/||b|| 5.414312147250e-03
>>    24 SNES Function norm 3.452679622612e+02 
>>       0 KSP preconditioned resid norm 6.149140486077e+03 true resid norm 3.452679622612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.400761945257e-01 true resid norm 3.795886928810e+00 ||r(i)||/||b|| 1.099403171945e-02
>>       2 KSP preconditioned resid norm 1.754461070240e-06 true resid norm 3.648059240080e+00 ||r(i)||/||b|| 1.056587821294e-02
>>    25 SNES Function norm 3.452667776779e+02 
>>       0 KSP preconditioned resid norm 1.101683177789e+04 true resid norm 3.452667776779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.317993935110e+00 true resid norm 2.454711939414e+01 ||r(i)||/||b|| 7.109609432809e-02
>>       2 KSP preconditioned resid norm 3.457477291813e-05 true resid norm 6.807469339003e+00 ||r(i)||/||b|| 1.971654899665e-02
>>    26 SNES Function norm 3.452666586474e+02 
>>       0 KSP preconditioned resid norm 3.879190850784e+04 true resid norm 3.452666586474e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.739562350939e+01 true resid norm 1.221587956319e+02 ||r(i)||/||b|| 3.538099974971e-01
>>       2 KSP preconditioned resid norm 7.455463420492e-05 true resid norm 3.080916578781e+01 ||r(i)||/||b|| 8.923295955800e-02
>>    27 SNES Function norm 3.452666425264e+02 
>>       0 KSP preconditioned resid norm 5.709318799171e+04 true resid norm 3.452666425264e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.292899448408e-01 true resid norm 2.801354737138e+01 ||r(i)||/||b|| 8.113597990933e-02
>>    28 SNES Function norm 3.452666374187e+02 
>>       0 KSP preconditioned resid norm 2.488913348294e+05 true resid norm 3.452666374187e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.691998534170e+02 true resid norm 7.244072115265e+03 ||r(i)||/||b|| 2.098109498625e+01
>>       2 KSP preconditioned resid norm 2.503929513070e-03 true resid norm 2.928791518636e+03 ||r(i)||/||b|| 8.482694825461e+00
>>    29 SNES Function norm 3.452666374177e+02 
>>       0 KSP preconditioned resid norm 2.533150641403e+05 true resid norm 3.452666374177e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.817060595281e+02 true resid norm 1.333479168158e+03 ||r(i)||/||b|| 3.862172082803e+00
>>       2 KSP preconditioned resid norm 8.930133492283e-04 true resid norm 5.412764197416e+02 ||r(i)||/||b|| 1.567705538507e+00
>>    30 SNES Function norm 3.452666373128e+02 
>>       0 KSP preconditioned resid norm 2.169409532487e+05 true resid norm 3.452666373128e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.325438416139e+02 true resid norm 9.441618139691e+02 ||r(i)||/||b|| 2.734587451940e+00
>>       2 KSP preconditioned resid norm 8.796591247860e-04 true resid norm 2.978002325996e+02 ||r(i)||/||b|| 8.625224693509e-01
>>    31 SNES Function norm 3.452666372449e+02 
>>       0 KSP preconditioned resid norm 1.459475274582e+05 true resid norm 3.452666372449e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.978987062939e+02 true resid norm 1.411576222768e+03 ||r(i)||/||b|| 4.088365542735e+00
>>       2 KSP preconditioned resid norm 1.235854958341e-03 true resid norm 3.579390387115e+02 ||r(i)||/||b|| 1.036703231936e+00
>>    32 SNES Function norm 3.452666369928e+02 
>>       0 KSP preconditioned resid norm 1.264234515603e+05 true resid norm 3.452666369928e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.676975931928e+01 true resid norm 1.855746501202e+02 ||r(i)||/||b|| 5.374821376792e-01
>>       2 KSP preconditioned resid norm 9.286140031498e-05 true resid norm 3.495715551553e+01 ||r(i)||/||b|| 1.012468387331e-01
>>    33 SNES Function norm 3.452666245901e+02 
>>       0 KSP preconditioned resid norm 1.544591438108e+06 true resid norm 3.452666245901e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.501087626485e+01 true resid norm 3.233291920369e+02 ||r(i)||/||b|| 9.364623424597e-01
>>       2 KSP preconditioned resid norm 1.867215855867e-04 true resid norm 7.114686634655e+01 ||r(i)||/||b|| 2.060635499623e-01
>>    34 SNES Function norm 3.452666216061e+02 
>>       0 KSP preconditioned resid norm 3.431570281248e+05 true resid norm 3.452666216061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.140176766342e+02 true resid norm 1.760366951993e+03 ||r(i)||/||b|| 5.098572644538e+00
>>       2 KSP preconditioned resid norm 2.324744486242e-03 true resid norm 1.080605579953e+03 ||r(i)||/||b|| 3.129771348664e+00
>>    35 SNES Function norm 3.452666215921e+02 
>>       0 KSP preconditioned resid norm 3.734063885343e+05 true resid norm 3.452666215921e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.279219133674e+00 true resid norm 1.691481367261e+02 ||r(i)||/||b|| 4.899058470990e-01
>>       2 KSP preconditioned resid norm 4.046466703016e-04 true resid norm 1.964786369469e+02 ||r(i)||/||b|| 5.690635139908e-01
>>    36 SNES Function norm 3.452666212200e+02 
>>       0 KSP preconditioned resid norm 7.952390829283e+05 true resid norm 3.452666212200e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.707607129733e+01 true resid norm 2.806640653978e+02 ||r(i)||/||b|| 8.128908158167e-01
>>       2 KSP preconditioned resid norm 1.296391885225e-04 true resid norm 1.079623998255e+02 ||r(i)||/||b|| 3.126928384910e-01
>>    37 SNES Function norm 3.452666199636e+02 
>>       0 KSP preconditioned resid norm 9.358626316710e+05 true resid norm 3.452666199636e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.722696571742e+01 true resid norm 3.251492717238e+02 ||r(i)||/||b|| 9.417338744130e-01
>>       2 KSP preconditioned resid norm 2.862108374145e-04 true resid norm 3.320104402606e+01 ||r(i)||/||b|| 9.616059620694e-02
>>    38 SNES Function norm 3.452666133867e+02 
>>       0 KSP preconditioned resid norm 8.365871848022e+04 true resid norm 3.452666133867e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.482543031390e+02 true resid norm 1.772891732407e+03 ||r(i)||/||b|| 5.134848443690e+00
>>       2 KSP preconditioned resid norm 1.439398367387e-03 true resid norm 3.088902452162e+02 ||r(i)||/||b|| 8.946426710257e-01
>>    39 SNES Function norm 3.452666132332e+02 
>>       0 KSP preconditioned resid norm 8.960112658041e+04 true resid norm 3.452666132332e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.505891399150e+01 true resid norm 1.787713293837e+02 ||r(i)||/||b|| 5.177776319280e-01
>>       2 KSP preconditioned resid norm 6.384856870578e-05 true resid norm 2.543851208517e+01 ||r(i)||/||b|| 7.367787996341e-02
>>    40 SNES Function norm 3.452665904125e+02 
>>       0 KSP preconditioned resid norm 7.796808144307e+05 true resid norm 3.452665904125e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.020764189760e+01 true resid norm 6.961545395677e+02 ||r(i)||/||b|| 2.016281212544e+00
>>       2 KSP preconditioned resid norm 3.506849883654e-04 true resid norm 4.956460306859e+02 ||r(i)||/||b|| 1.435545878024e+00
>>    41 SNES Function norm 3.452665902941e+02 
>>       0 KSP preconditioned resid norm 4.990797566798e+05 true resid norm 3.452665902941e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.012324354359e+02 true resid norm 7.390530765626e+02 ||r(i)||/||b|| 2.140528789458e+00
>>       2 KSP preconditioned resid norm 7.111862680292e-04 true resid norm 4.066260265930e+02 ||r(i)||/||b|| 1.177716112777e+00
>>    42 SNES Function norm 3.452665901159e+02 
>>       0 KSP preconditioned resid norm 3.472654025442e+05 true resid norm 3.452665901159e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.552680791160e+02 true resid norm 1.064105325830e+03 ||r(i)||/||b|| 3.081981739016e+00
>>       2 KSP preconditioned resid norm 2.695346315896e-03 true resid norm 5.359619569435e+02 ||r(i)||/||b|| 1.552313407340e+00
>>    43 SNES Function norm 3.452665900083e+02 
>>       0 KSP preconditioned resid norm 2.823873507449e+05 true resid norm 3.452665900083e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.616335635707e+00 true resid norm 1.564312339233e+02 ||r(i)||/||b|| 4.530737651723e-01
>>       2 KSP preconditioned resid norm 1.580836053583e-04 true resid norm 1.777870114826e+02 ||r(i)||/||b|| 5.149267743465e-01
>>    44 SNES Function norm 3.452665895210e+02 
>>       0 KSP preconditioned resid norm 4.764215602672e+05 true resid norm 3.452665895210e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.491949190147e+01 true resid norm 4.652947485976e+02 ||r(i)||/||b|| 1.347639078670e+00
>>       2 KSP preconditioned resid norm 2.266106888017e-04 true resid norm 1.316722015429e+02 ||r(i)||/||b|| 3.813638664707e-01
>>    45 SNES Function norm 3.452665889075e+02 
>>       0 KSP preconditioned resid norm 1.439659890301e+05 true resid norm 3.452665889075e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.431461947162e+01 true resid norm 1.403614551576e+02 ||r(i)||/||b|| 4.065306625865e-01
>>       2 KSP preconditioned resid norm 4.711117728059e-04 true resid norm 1.095117712535e+02 ||r(i)||/||b|| 3.171803318704e-01
>>    46 SNES Function norm 3.452665876547e+02 
>>       0 KSP preconditioned resid norm 2.157313334766e+05 true resid norm 3.452665876547e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.967750174401e+00 true resid norm 1.053591658730e+02 ||r(i)||/||b|| 3.051530893525e-01
>>       2 KSP preconditioned resid norm 4.512059841816e-04 true resid norm 1.189061020519e+02 ||r(i)||/||b|| 3.443892525473e-01
>>    47 SNES Function norm 3.452665866229e+02 
>>       0 KSP preconditioned resid norm 3.981118078251e+05 true resid norm 3.452665866229e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.013119377232e+01 true resid norm 4.969789311062e+02 ||r(i)||/||b|| 1.439406390196e+00
>>       2 KSP preconditioned resid norm 1.872910444347e-04 true resid norm 1.396396719776e+02 ||r(i)||/||b|| 4.044401554852e-01
>>    48 SNES Function norm 3.452665850954e+02 
>>       0 KSP preconditioned resid norm 1.967554630062e+05 true resid norm 3.452665850954e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.006970327024e+00 true resid norm 9.616098906643e+01 ||r(i)||/||b|| 2.785122951874e-01
>>       2 KSP preconditioned resid norm 3.184368801558e-05 true resid norm 9.951979756268e+01 ||r(i)||/||b|| 2.882404549377e-01
>>    49 SNES Function norm 3.452665835600e+02 
>>       0 KSP preconditioned resid norm 3.965648247310e+05 true resid norm 3.452665835600e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.862969730098e+01 true resid norm 1.790616593203e+02 ||r(i)||/||b|| 5.186185627174e-01
>>       2 KSP preconditioned resid norm 2.964457441143e-04 true resid norm 1.447818294544e+02 ||r(i)||/||b|| 4.193334552146e-01
>>    50 SNES Function norm 3.452665828612e+02 
>>       0 KSP preconditioned resid norm 1.265022286683e+06 true resid norm 3.452665828612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.401066920177e+01 true resid norm 3.143671803790e+02 ||r(i)||/||b|| 9.105056671685e-01
>>       2 KSP preconditioned resid norm 1.444898193175e-04 true resid norm 7.578180059616e+01 ||r(i)||/||b|| 2.194877939480e-01
>>    51 SNES Function norm 3.452665817118e+02 
>>       0 KSP preconditioned resid norm 2.063334812849e+05 true resid norm 3.452665817118e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.074628697499e+01 true resid norm 1.155569679685e+02 ||r(i)||/||b|| 3.346891187543e-01
>>       2 KSP preconditioned resid norm 1.326420665890e-04 true resid norm 9.261090091148e+01 ||r(i)||/||b|| 2.682301323584e-01
>>    52 SNES Function norm 3.452665807811e+02 
>>       0 KSP preconditioned resid norm 6.549963238965e+06 true resid norm 3.452665807811e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.246352248726e+01 true resid norm 3.604213542889e+02 ||r(i)||/||b|| 1.043892963731e+00
>>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 52
>> 
>> which is a little different, but not so much different, than what i get if I use my Jacobian.
>> 
>> -gideon
>> 
>>> On Sep 8, 2015, at 10:00 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> Yup, now it?s working.
>>> -gideon
>>> 
>>>> On Sep 8, 2015, at 9:58 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>> 
>>>> On Tue, Sep 8, 2015 at 8:51 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> Going back to this point, a case that previously worked, is now giving me problems that I don?t understand:
>>>> 
>>>>   0 SNES Function norm 1.421454390131e-02 
>>>>     0 KSP preconditioned resid norm 7.635339584553e-03 true resid norm 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.838956718317e-07 true resid norm 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
>>>>     2 KSP preconditioned resid norm 5.889748199467e-12 true resid norm 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
>>>>   1 SNES Function norm 4.476139939604e-06 
>>>>     0 KSP preconditioned resid norm 4.897102387844e-05 true resid norm 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.133649519960e-09 true resid norm 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
>>>>     2 KSP preconditioned resid norm 2.223328042462e-14 true resid norm 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
>>>>   2 SNES Function norm 5.447761863330e-09 
>>>>     0 KSP preconditioned resid norm 9.703890346186e-09 true resid norm 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.982949185218e-13 true resid norm 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
>>>>     2 KSP preconditioned resid norm 7.045835741315e-18 true resid norm 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
>>>> Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT iterations 2
>>>> 
>>>> Somehow the maximum number of function evaluations is being exceeded. Oh, are you using -snes_fd? This uses a function evaluation
>>>> for each column, so you might have to up it > 10000. The convergence is fine.
>>>> 
>>>>   Thanks,
>>>> 
>>>>      Matt
>>>>  
>>>> -gideon
>>>> 
>>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>>>> 
>>>>>  Barry
>>>>> 
>>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>> 
>>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>>>>> 
>>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>>>> 
>>>>>>>   Matt
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>>>> 
>>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>>>> 
>>>>>>>> Gideon,
>>>>>>>> 
>>>>>>>>  In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>>>> 
>>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>>>>> 
>>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>>>> 
>>>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>>>> 
>>>>>>>>>>>   0 SNES Function norm 3.857327250267e-09 
>>>>>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>>>   1 SNES Function norm 2.479697868412e-12 
>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>>>> 
>>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>>>> 
>>>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>  0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>>>>  1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> then 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>>>>    type: seqaij
>>>>>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>>>      MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>>>>        PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>>>>          [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0]  143924 
>>>>>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0] 15991 
>>>>>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>>>    SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>>>    etc
>>>>>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From gideon.simpson at gmail.com  Tue Sep  8 22:00:43 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 23:00:43 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
Message-ID: <65B2C6D5-47C9-49A1-9025-283168A07A0A@gmail.com>

Ok, again, I?m ignorant, where do I put the VecView command to get it to save at some particular iterate?  Also, what do you mean by ?halves??


-gideon

> On Sep 8, 2015, at 10:53 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Sep 8, 2015 at 9:48 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> Your comment about the different pieces scaling different made me think of something, and I made a slight modification.  It still fails, but the conditioning looks much better, no?
> 
> This is great. Now its clear that the nonlinear convergence is crappy.
> 
> Lets start with looking at the distribution of residuals. Take iterate 50,
> which is as converged as you get and save the residual (VecView with
> binary). Then we can load it up and play with it. I recommend
> 
>   a) Norms for both halves
> 
>   b) Plot of the big half
> 
> What usually happens is that some small part is screwing up, but you
> have to iterate the whole system, and finding the descent direction is
> hard. If you select the problem variables, Newton will converge, and then
> the big system will converge once the problem is removed. This is the classic
> "coarse basis selection" problem in the nonlinear regime.
> 
>    Matt
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/c0adabee/attachment.html>

From gideon.simpson at gmail.com  Tue Sep  8 22:28:08 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 8 Sep 2015 23:28:08 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
Message-ID: <78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>

I should clarify, in that run, I ran with the analytic jacobian, but I did not use -snes_mf_operator.  If I use that flag, the performance is a bit different.  In particular, the true residual norms are not as good.  How should I interpret that?

      0 SNES Function norm 1.975693696426e-08 
        0 KSP preconditioned resid norm 4.241029862559e-08 true resid norm 1.975693696426e-08 ||r(i)||/||b|| 1.000000000000e+00
        1 KSP preconditioned resid norm 8.429547895868e-14 true resid norm 1.953386034503e-12 ||r(i)||/||b|| 9.887089471596e-05
      1 SNES Function norm 2.220439366420e-12 
    Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
    0 SNES Function norm 5.066222213175e+03 
      0 KSP preconditioned resid norm 3.135087050116e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.763659023735e-04 true resid norm 2.125311397199e+00 ||r(i)||/||b|| 4.195061542448e-04
      2 KSP preconditioned resid norm 1.084917879345e-09 true resid norm 9.170494798321e-03 ||r(i)||/||b|| 1.810124864731e-06
    1 SNES Function norm 8.488370402237e+02 
      0 KSP preconditioned resid norm 2.671196814418e+02 true resid norm 8.488370402237e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.493447352396e-03 true resid norm 1.977751087754e-01 ||r(i)||/||b|| 2.329953800358e-04
      2 KSP preconditioned resid norm 2.764181028108e-08 true resid norm 1.635196652391e-01 ||r(i)||/||b|| 1.926396439957e-04
    2 SNES Function norm 6.550793737252e+02 
      0 KSP preconditioned resid norm 1.534706412676e+02 true resid norm 6.550793737252e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.638157030055e-03 true resid norm 8.038238857445e-02 ||r(i)||/||b|| 1.227063342223e-04
      2 KSP preconditioned resid norm 4.712628487080e-08 true resid norm 7.077842320372e-02 ||r(i)||/||b|| 1.080455682816e-04
    3 SNES Function norm 5.773378779059e+02 
      0 KSP preconditioned resid norm 1.357060892902e+02 true resid norm 5.773378779059e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.956356650328e-02 true resid norm 6.547565039272e-01 ||r(i)||/||b|| 1.134095871731e-03
      2 KSP preconditioned resid norm 4.061258280947e-08 true resid norm 6.285340893993e-02 ||r(i)||/||b|| 1.088676342663e-04
    4 SNES Function norm 5.240279637579e+02 
      0 KSP preconditioned resid norm 1.520925023264e+02 true resid norm 5.240279637579e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.439871073884e-02 true resid norm 7.370842666984e-01 ||r(i)||/||b|| 1.406574300754e-03
      2 KSP preconditioned resid norm 2.929266069899e-08 true resid norm 6.966469677574e-02 ||r(i)||/||b|| 1.329408000981e-04
    5 SNES Function norm 4.757123720235e+02 
      0 KSP preconditioned resid norm 1.874598394237e+02 true resid norm 4.757123720235e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.666900422546e-02 true resid norm 4.366093312686e-01 ||r(i)||/||b|| 9.178010851630e-04
      2 KSP preconditioned resid norm 1.569055410459e-07 true resid norm 8.415473672611e-02 ||r(i)||/||b|| 1.769025606127e-04
    6 SNES Function norm 4.225299506849e+02 
      0 KSP preconditioned resid norm 2.568609427837e+02 true resid norm 4.225299506849e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.408384520223e-02 true resid norm 5.711566149678e-01 ||r(i)||/||b|| 1.351754151491e-03
      2 KSP preconditioned resid norm 1.091907927355e-07 true resid norm 1.205574809113e-01 ||r(i)||/||b|| 2.853229237735e-04
    7 SNES Function norm 3.809903859846e+02 
      0 KSP preconditioned resid norm 5.148606473696e+02 true resid norm 3.809903859846e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.519554844371e-02 true resid norm 5.230376280503e-01 ||r(i)||/||b|| 1.372836815025e-03
      2 KSP preconditioned resid norm 7.972364126927e-07 true resid norm 2.637500099868e-01 ||r(i)||/||b|| 6.922747126680e-04
    8 SNES Function norm 3.759602001862e+02 
      0 KSP preconditioned resid norm 5.342247036495e+03 true resid norm 3.759602001862e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.133750669625e+00 true resid norm 1.703127877004e+01 ||r(i)||/||b|| 4.530074928571e-02
      2 KSP preconditioned resid norm 4.996777563109e-06 true resid norm 2.867628968895e+00 ||r(i)||/||b|| 7.627480162728e-03
    9 SNES Function norm 3.756183831703e+02 
      0 KSP preconditioned resid norm 5.800146189851e+03 true resid norm 3.756183831703e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.928446833882e-01 true resid norm 6.567210094215e+00 ||r(i)||/||b|| 1.748372920086e-02
      2 KSP preconditioned resid norm 3.716020308248e-06 true resid norm 3.211789561370e+00 ||r(i)||/||b|| 8.550671919361e-03
   10 SNES Function norm 3.746165124359e+02 
      0 KSP preconditioned resid norm 6.573797317895e+02 true resid norm 3.746165124359e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.168407014568e-01 true resid norm 1.224122498177e+00 ||r(i)||/||b|| 3.267668288879e-03
      2 KSP preconditioned resid norm 6.626961415306e-07 true resid norm 3.840346541352e-01 ||r(i)||/||b|| 1.025140754309e-03
   11 SNES Function norm 3.721805808051e+02 
      0 KSP preconditioned resid norm 7.849993923903e+02 true resid norm 3.721805808051e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.857279790238e-02 true resid norm 5.226376723903e-01 ||r(i)||/||b|| 1.404258307244e-03
      2 KSP preconditioned resid norm 6.283315938786e-07 true resid norm 4.622839625158e-01 ||r(i)||/||b|| 1.242095870547e-03
   12 SNES Function norm 3.702012231848e+02 
      0 KSP preconditioned resid norm 8.145587172693e+03 true resid norm 3.702012231848e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.293173409305e+00 true resid norm 1.444669645437e+01 ||r(i)||/||b|| 3.902390254166e-02
      2 KSP preconditioned resid norm 5.333263851161e-06 true resid norm 4.631558046987e+00 ||r(i)||/||b|| 1.251092043171e-02
   13 SNES Function norm 3.701675832926e+02 
      0 KSP preconditioned resid norm 3.439618250319e+03 true resid norm 3.701675832926e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.424584259775e-01 true resid norm 9.488954827528e+00 ||r(i)||/||b|| 2.563421340984e-02
      2 KSP preconditioned resid norm 2.361993681793e-06 true resid norm 2.151423050784e+00 ||r(i)||/||b|| 5.812024466453e-03
   14 SNES Function norm 3.700043773583e+02 
      0 KSP preconditioned resid norm 1.591977430573e+03 true resid norm 3.700043773583e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.627624143289e-01 true resid norm 1.937528552277e+00 ||r(i)||/||b|| 5.236501703332e-03
      2 KSP preconditioned resid norm 1.406173142258e-06 true resid norm 9.301779435764e-01 ||r(i)||/||b|| 2.513964700141e-03
   15 SNES Function norm 3.694535945999e+02 
      0 KSP preconditioned resid norm 1.121321894103e+03 true resid norm 3.694535945999e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.897150400722e-01 true resid norm 2.095277723848e+00 ||r(i)||/||b|| 5.671287962747e-03
      2 KSP preconditioned resid norm 7.510763366789e-07 true resid norm 6.477051057041e-01 ||r(i)||/||b|| 1.753143331588e-03
   16 SNES Function norm 3.678972879038e+02 
      0 KSP preconditioned resid norm 3.333082468735e+01 true resid norm 3.678972879038e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.431732430685e-03 true resid norm 1.616512305791e-02 ||r(i)||/||b|| 4.393922866356e-05
      2 KSP preconditioned resid norm 6.434957827496e-08 true resid norm 1.918034542828e-02 ||r(i)||/||b|| 5.213505524211e-05
   17 SNES Function norm 3.342697499638e+02 
      0 KSP preconditioned resid norm 1.247233383993e+02 true resid norm 3.342697499638e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 5.479213016905e-02 true resid norm 2.646077519018e-01 ||r(i)||/||b|| 7.915994550225e-04
      2 KSP preconditioned resid norm 1.491221634262e-07 true resid norm 7.551908725240e-02 ||r(i)||/||b|| 2.259225887493e-04
   18 SNES Function norm 3.242724990245e+02 
      0 KSP preconditioned resid norm 2.730129105212e+02 true resid norm 3.242724990245e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.298117117474e-02 true resid norm 2.331723933868e-01 ||r(i)||/||b|| 7.190631153991e-04
      2 KSP preconditioned resid norm 2.245058389453e-07 true resid norm 1.607094606122e-01 ||r(i)||/||b|| 4.956000311333e-04
   19 SNES Function norm 3.161198940824e+02 
      0 KSP preconditioned resid norm 1.976453347207e+02 true resid norm 3.161198940824e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.069966316351e-02 true resid norm 3.606207026495e-01 ||r(i)||/||b|| 1.140771933055e-03
      2 KSP preconditioned resid norm 1.339731244693e-06 true resid norm 1.150564760533e-01 ||r(i)||/||b|| 3.639646798796e-04
   20 SNES Function norm 3.095294698084e+02 
      0 KSP preconditioned resid norm 1.989229053011e+02 true resid norm 3.095294698084e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.509913758989e-02 true resid norm 3.251999852245e-01 ||r(i)||/||b|| 1.050626893219e-03
      2 KSP preconditioned resid norm 3.119014445967e-07 true resid norm 1.169978920551e-01 ||r(i)||/||b|| 3.779862774539e-04
   21 SNES Function norm 3.041156286651e+02 
      0 KSP preconditioned resid norm 6.280117878327e+02 true resid norm 3.041156286651e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.168409567185e-01 true resid norm 1.048814657078e+00 ||r(i)||/||b|| 3.448736461464e-03
      2 KSP preconditioned resid norm 1.609317462605e-06 true resid norm 3.730072311447e-01 ||r(i)||/||b|| 1.226530950684e-03
   22 SNES Function norm 3.040366549628e+02 
      0 KSP preconditioned resid norm 1.437979588458e+03 true resid norm 3.040366549628e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.585745252784e-01 true resid norm 3.748888651074e+00 ||r(i)||/||b|| 1.233038382011e-02
      2 KSP preconditioned resid norm 2.280617561277e-06 true resid norm 8.938758394240e-01 ||r(i)||/||b|| 2.940026555460e-03
   23 SNES Function norm 3.038811444081e+02 
      0 KSP preconditioned resid norm 4.656588555628e+02 true resid norm 3.038811444081e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.379807174952e-01 true resid norm 1.502298425690e+00 ||r(i)||/||b|| 4.943703988664e-03
      2 KSP preconditioned resid norm 1.371459645956e-06 true resid norm 2.736849872814e-01 ||r(i)||/||b|| 9.006316855046e-04
   24 SNES Function norm 3.021353442803e+02 
      0 KSP preconditioned resid norm 2.323079380432e+02 true resid norm 3.021353442803e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.154620938700e-01 true resid norm 1.429767871615e+00 ||r(i)||/||b|| 4.732209914140e-03
      2 KSP preconditioned resid norm 7.092074512403e-07 true resid norm 1.370959604626e-01 ||r(i)||/||b|| 4.537567784038e-04
   25 SNES Function norm 2.989453281461e+02 
      0 KSP preconditioned resid norm 2.466046529201e+02 true resid norm 2.989453281461e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.542626606314e-01 true resid norm 1.672331292335e+00 ||r(i)||/||b|| 5.594104121668e-03
      2 KSP preconditioned resid norm 8.410454494090e-07 true resid norm 1.432702712514e-01 ||r(i)||/||b|| 4.792524176238e-04
   26 SNES Function norm 2.965575466120e+02 
      0 KSP preconditioned resid norm 1.729310576069e+02 true resid norm 2.965575466120e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.667588027964e-02 true resid norm 9.915795006833e-01 ||r(i)||/||b|| 3.343632667628e-03
      2 KSP preconditioned resid norm 5.726421091521e-07 true resid norm 1.042312787918e-01 ||r(i)||/||b|| 3.514706672706e-04
   27 SNES Function norm 2.886519975216e+02 
      0 KSP preconditioned resid norm 3.474883023572e+02 true resid norm 2.886519975216e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.449042223312e-01 true resid norm 2.910159944511e+00 ||r(i)||/||b|| 1.008189781986e-02
      2 KSP preconditioned resid norm 4.054998693233e-06 true resid norm 2.177883931275e-01 ||r(i)||/||b|| 7.545015970699e-04
   28 SNES Function norm 2.877765511463e+02 
      0 KSP preconditioned resid norm 6.932304384488e+02 true resid norm 2.877765511463e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.623309918707e-01 true resid norm 6.631612349599e+00 ||r(i)||/||b|| 2.304431102251e-02
      2 KSP preconditioned resid norm 8.935815040540e-06 true resid norm 4.208867273501e-01 ||r(i)||/||b|| 1.462546985408e-03
   29 SNES Function norm 2.874829836383e+02 
      0 KSP preconditioned resid norm 3.031373648629e+02 true resid norm 2.874829836383e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.097809781088e-01 true resid norm 8.269361024084e-01 ||r(i)||/||b|| 2.876469737244e-03
      2 KSP preconditioned resid norm 1.242142566726e-06 true resid norm 1.780035410476e-01 ||r(i)||/||b|| 6.191793990544e-04
   30 SNES Function norm 2.857592509565e+02 
      0 KSP preconditioned resid norm 2.905142832215e+02 true resid norm 2.857592509565e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.993930292806e-02 true resid norm 4.363268013941e-01 ||r(i)||/||b|| 1.526903503329e-03
      2 KSP preconditioned resid norm 1.565987527263e-06 true resid norm 1.752909971699e-01 ||r(i)||/||b|| 6.134219507616e-04
   31 SNES Function norm 2.835404531301e+02 
      0 KSP preconditioned resid norm 1.787845486677e+02 true resid norm 2.835404531301e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.133510370229e-01 true resid norm 1.954968834465e+00 ||r(i)||/||b|| 6.894849792627e-03
      2 KSP preconditioned resid norm 1.325071802836e-06 true resid norm 1.055727452178e-01 ||r(i)||/||b|| 3.723375061738e-04
   32 SNES Function norm 2.766251793953e+02 
      0 KSP preconditioned resid norm 8.373384127615e+01 true resid norm 2.766251793953e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.599342282394e-02 true resid norm 1.296081686745e-01 ||r(i)||/||b|| 4.685335187412e-04
      2 KSP preconditioned resid norm 8.378945187131e-08 true resid norm 4.905876720151e-02 ||r(i)||/||b|| 1.773474392633e-04
   33 SNES Function norm 2.763417748035e+02 
      0 KSP preconditioned resid norm 4.315380927138e+02 true resid norm 2.763417748035e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.240956674451e-02 true resid norm 8.440426813509e-01 ||r(i)||/||b|| 3.054343419307e-03
      2 KSP preconditioned resid norm 1.055259274458e-06 true resid norm 2.556138363110e-01 ||r(i)||/||b|| 9.249916574963e-04
   34 SNES Function norm 2.636632257368e+02 
      0 KSP preconditioned resid norm 2.002447589825e+02 true resid norm 2.636632257368e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.091871038526e-01 true resid norm 6.938948038720e-01 ||r(i)||/||b|| 2.631746622734e-03
      2 KSP preconditioned resid norm 1.078660902246e-06 true resid norm 1.161053485203e-01 ||r(i)||/||b|| 4.403547297725e-04
   35 SNES Function norm 2.607600958607e+02 
      0 KSP preconditioned resid norm 1.036034642027e+02 true resid norm 2.607600958607e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.431459913062e-02 true resid norm 2.275229425112e-01 ||r(i)||/||b|| 8.725374247171e-04
      2 KSP preconditioned resid norm 2.358846414843e-07 true resid norm 6.052444521984e-02 ||r(i)||/||b|| 2.321077733158e-04
   36 SNES Function norm 2.559001581389e+02 
      0 KSP preconditioned resid norm 1.549260221638e+02 true resid norm 2.559001581389e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.556726070406e-01 true resid norm 1.861290598804e+00 ||r(i)||/||b|| 7.273503120674e-03
      2 KSP preconditioned resid norm 1.968725491683e-06 true resid norm 9.363655703756e-02 ||r(i)||/||b|| 3.659105086865e-04
   37 SNES Function norm 2.518397862326e+02 
      0 KSP preconditioned resid norm 5.597954308908e+01 true resid norm 2.518397862326e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.026654037349e-02 true resid norm 1.814101931154e-01 ||r(i)||/||b|| 7.203396882964e-04
      2 KSP preconditioned resid norm 6.253768805342e-08 true resid norm 3.229917855960e-02 ||r(i)||/||b|| 1.282528826870e-04
   38 SNES Function norm 2.316134843390e+02 
      0 KSP preconditioned resid norm 7.012435360694e+02 true resid norm 2.316134843390e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.076320346419e-01 true resid norm 4.417669735438e+00 ||r(i)||/||b|| 1.907345657377e-02
      2 KSP preconditioned resid norm 1.250819622244e-06 true resid norm 4.042382807831e-01 ||r(i)||/||b|| 1.745314103524e-03
   39 SNES Function norm 2.315003516125e+02 
      0 KSP preconditioned resid norm 1.770628877633e+03 true resid norm 2.315003516125e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.699599407714e-01 true resid norm 5.719451377836e+00 ||r(i)||/||b|| 2.470601594337e-02
      2 KSP preconditioned resid norm 1.342291667594e-05 true resid norm 1.009488471262e+00 ||r(i)||/||b|| 4.360634721416e-03
   40 SNES Function norm 2.314745966456e+02 
      0 KSP preconditioned resid norm 3.814990651391e+03 true resid norm 2.314745966456e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 5.487459394941e+00 true resid norm 3.985946817773e+01 ||r(i)||/||b|| 1.721980241260e-01
      2 KSP preconditioned resid norm 1.472896798351e-05 true resid norm 2.616262470753e+00 ||r(i)||/||b|| 1.130259004083e-02
   41 SNES Function norm 2.314707133124e+02 
      0 KSP preconditioned resid norm 3.619965563807e+04 true resid norm 2.314707133124e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.046216042689e+01 true resid norm 1.485715627898e+02 ||r(i)||/||b|| 6.418590095641e-01
      2 KSP preconditioned resid norm 5.088996886285e-05 true resid norm 7.571617793696e+00 ||r(i)||/||b|| 3.271091053095e-02
   42 SNES Function norm 2.314704913770e+02 
      0 KSP preconditioned resid norm 6.684430447231e+03 true resid norm 2.314704913770e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.035066002617e+00 true resid norm 1.498611774406e+01 ||r(i)||/||b|| 6.474310247889e-02
      2 KSP preconditioned resid norm 1.894517227201e-05 true resid norm 3.620274714957e+00 ||r(i)||/||b|| 1.564032932846e-02
   43 SNES Function norm 2.314685063519e+02 
      0 KSP preconditioned resid norm 1.519862455568e+04 true resid norm 2.314685063519e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.799545478657e+02 true resid norm 1.315130172649e+03 ||r(i)||/||b|| 5.681680818597e+00
      2 KSP preconditioned resid norm 7.538897078155e-04 true resid norm 5.814357004150e+01 ||r(i)||/||b|| 2.511943026629e-01
   44 SNES Function norm 2.314684987281e+02 
      0 KSP preconditioned resid norm 1.914858613894e+04 true resid norm 2.314684987281e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.610361797677e+00 true resid norm 2.904545821866e+01 ||r(i)||/||b|| 1.254834173041e-01
      2 KSP preconditioned resid norm 8.459090941337e-06 true resid norm 9.804924579099e+00 ||r(i)||/||b|| 4.235964994363e-02
   45 SNES Function norm 2.314682724575e+02 
      0 KSP preconditioned resid norm 1.628064355012e+04 true resid norm 2.314682724575e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.192253277458e+01 true resid norm 8.717348030044e+01 ||r(i)||/||b|| 3.766109254409e-01
      2 KSP preconditioned resid norm 2.901075005128e-05 true resid norm 5.898778957654e+00 ||r(i)||/||b|| 2.548417930037e-02
   46 SNES Function norm 2.314675179712e+02 
      0 KSP preconditioned resid norm 1.393728659990e+04 true resid norm 2.314675179712e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.363450892258e+01 true resid norm 9.876887689991e+01 ||r(i)||/||b|| 4.267072882002e-01
      2 KSP preconditioned resid norm 3.348633706956e-05 true resid norm 4.651974796776e+00 ||r(i)||/||b|| 2.009774346548e-02
   47 SNES Function norm 2.314663411831e+02 
      0 KSP preconditioned resid norm 7.856043279131e+03 true resid norm 2.314663411831e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.677460009854e+00 true resid norm 2.021525646408e+01 ||r(i)||/||b|| 8.733562020616e-02
      2 KSP preconditioned resid norm 4.033484713601e-05 true resid norm 4.191927270226e+00 ||r(i)||/||b|| 1.811031033194e-02
   48 SNES Function norm 2.314648869203e+02 
      0 KSP preconditioned resid norm 1.052613791778e+04 true resid norm 2.314648869203e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.177731181359e+00 true resid norm 2.344482867012e+01 ||r(i)||/||b|| 1.012889211061e-01
      2 KSP preconditioned resid norm 4.337358847947e-05 true resid norm 5.365943818938e+00 ||r(i)||/||b|| 2.318253921937e-02
   49 SNES Function norm 2.314640197245e+02 
      0 KSP preconditioned resid norm 2.047116917526e+04 true resid norm 2.314640197245e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.560093775943e+00 true resid norm 7.015143565198e+01 ||r(i)||/||b|| 3.030770645713e-01
      2 KSP preconditioned resid norm 5.024217804080e-05 true resid norm 8.482232519276e+00 ||r(i)||/||b|| 3.664600886726e-02
   50 SNES Function norm 2.314636382129e+02 
      0 KSP preconditioned resid norm 2.202428764601e+04 true resid norm 2.314636382129e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.644800264238e+02 true resid norm 2.634313227181e+03 ||r(i)||/||b|| 1.138111043065e+01
      2 KSP preconditioned resid norm 2.914035192146e-03 true resid norm 1.866099396017e+02 ||r(i)||/||b|| 8.062170846464e-01
   51 SNES Function norm 2.314636367493e+02 
      0 KSP preconditioned resid norm 1.948632462873e+04 true resid norm 2.314636367493e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.300034176850e+01 true resid norm 9.512460873903e+01 ||r(i)||/||b|| 4.109699911182e-01
      2 KSP preconditioned resid norm 5.189046693053e-05 true resid norm 6.633939889168e+00 ||r(i)||/||b|| 2.866082976288e-02
   52 SNES Function norm 2.314630481752e+02 
      0 KSP preconditioned resid norm 1.491442414085e+04 true resid norm 2.314630481752e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.379241745164e+01 true resid norm 1.007276964292e+02 ||r(i)||/||b|| 4.351783026420e-01
      2 KSP preconditioned resid norm 4.161679438843e-05 true resid norm 4.977179034848e+00 ||r(i)||/||b|| 2.150312576494e-02
   53 SNES Function norm 2.314619494854e+02 
      0 KSP preconditioned resid norm 1.072804170247e+04 true resid norm 2.314619494854e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.968714022053e+00 true resid norm 1.493125196687e+01 ||r(i)||/||b|| 6.450845160541e-02
      2 KSP preconditioned resid norm 9.504384832756e-06 true resid norm 5.788047965007e+00 ||r(i)||/||b|| 2.500647721094e-02
   54 SNES Function norm 2.314611811294e+02 
      0 KSP preconditioned resid norm 2.352264129011e+04 true resid norm 2.314611811294e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.347783610050e+01 true resid norm 9.827481962397e+01 ||r(i)||/||b|| 4.245844557797e-01
      2 KSP preconditioned resid norm 2.918515899364e-05 true resid norm 7.891003112752e+00 ||r(i)||/||b|| 3.409212324177e-02
   55 SNES Function norm 2.314607568296e+02 
      0 KSP preconditioned resid norm 1.749283175470e+04 true resid norm 2.314607568296e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.070162653778e+00 true resid norm 2.389094964330e+01 ||r(i)||/||b|| 1.032181436307e-01
      2 KSP preconditioned resid norm 1.631144493664e-05 true resid norm 1.139801540267e+01 ||r(i)||/||b|| 4.924383536454e-02
   56 SNES Function norm 2.314605438312e+02 
      0 KSP preconditioned resid norm 7.109914723607e+04 true resid norm 2.314605438312e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.363804435455e+01 true resid norm 6.067805272857e+02 ||r(i)||/||b|| 2.621528996874e+00
      2 KSP preconditioned resid norm 3.256368262965e-04 true resid norm 9.390634078085e+01 ||r(i)||/||b|| 4.057120890952e-01
   57 SNES Function norm 2.314605414352e+02 
      0 KSP preconditioned resid norm 2.888647244791e+04 true resid norm 2.314605414352e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.096880192515e+01 true resid norm 8.143424079389e+01 ||r(i)||/||b|| 3.518277469194e-01
      2 KSP preconditioned resid norm 8.022062493403e-05 true resid norm 1.077904577963e+01 ||r(i)||/||b|| 4.656969050877e-02
   58 SNES Function norm 2.314603336141e+02 
      0 KSP preconditioned resid norm 1.836068875705e+04 true resid norm 2.314603336141e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 9.090981797341e+01 true resid norm 6.587841779043e+02 ||r(i)||/||b|| 2.846207674628e+00
      2 KSP preconditioned resid norm 8.275701701453e-04 true resid norm 4.183336915180e+01 ||r(i)||/||b|| 1.807366666184e-01
   59 SNES Function norm 2.314603179098e+02 
      0 KSP preconditioned resid norm 2.853255155449e+04 true resid norm 2.314603179098e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.327796718536e+01 true resid norm 4.583228284114e+02 ||r(i)||/||b|| 1.980135655866e+00
      2 KSP preconditioned resid norm 5.951560456364e-04 true resid norm 5.003183584303e+01 ||r(i)||/||b|| 2.161572933747e-01
   60 SNES Function norm 2.314603080054e+02 
      0 KSP preconditioned resid norm 7.163155165743e+04 true resid norm 2.314603080054e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.633016255887e+01 true resid norm 3.379504724720e+02 ||r(i)||/||b|| 1.460079593708e+00
      2 KSP preconditioned resid norm 1.403149036303e-04 true resid norm 2.609855476909e+01 ||r(i)||/||b|| 1.127560703344e-01
   61 SNES Function norm 2.314602358658e+02 
      0 KSP preconditioned resid norm 1.802787930999e+04 true resid norm 2.314602358658e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.381933291542e+01 true resid norm 1.003023231896e+02 ||r(i)||/||b|| 4.333458091167e-01
      2 KSP preconditioned resid norm 1.089082641399e-04 true resid norm 5.919741402584e+00 ||r(i)||/||b|| 2.557563021761e-02
   62 SNES Function norm 2.314594914718e+02 
      0 KSP preconditioned resid norm 1.293341944019e+04 true resid norm 2.314594914718e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.507379326494e+00 true resid norm 1.900387953811e+01 ||r(i)||/||b|| 8.210455927845e-02
      2 KSP preconditioned resid norm 1.049822021718e-05 true resid norm 8.049347657678e+00 ||r(i)||/||b|| 3.477648553746e-02
   63 SNES Function norm 2.314590886372e+02 
      0 KSP preconditioned resid norm 5.233660621212e+04 true resid norm 2.314590886372e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.767828625949e+01 true resid norm 4.972067876731e+02 ||r(i)||/||b|| 2.148141127664e+00
      2 KSP preconditioned resid norm 1.565714280046e-03 true resid norm 9.541512803665e+01 ||r(i)||/||b|| 4.122332313604e-01
   64 SNES Function norm 2.314590857486e+02 
      0 KSP preconditioned resid norm 9.223438894260e+04 true resid norm 2.314590857486e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.771340682426e+01 true resid norm 1.439668879180e+02 ||r(i)||/||b|| 6.219971337587e-01
      2 KSP preconditioned resid norm 4.313497493249e-04 true resid norm 8.319281395963e+01 ||r(i)||/||b|| 3.594277307826e-01
   65 SNES Function norm 2.314590820052e+02 
      0 KSP preconditioned resid norm 2.716078594523e+06 true resid norm 2.314590820052e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.132140332876e+01 true resid norm 2.279043369448e+02 ||r(i)||/||b|| 9.846420152120e-01
      2 KSP preconditioned resid norm 3.421563776706e-04 true resid norm 3.596175647535e+01 ||r(i)||/||b|| 1.553698224489e-01
   66 SNES Function norm 2.314590616296e+02 
      0 KSP preconditioned resid norm 4.202813913686e+04 true resid norm 2.314590616296e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.895156465866e+00 true resid norm 5.779056270257e+01 ||r(i)||/||b|| 2.496794132651e-01
      2 KSP preconditioned resid norm 1.534565623678e-04 true resid norm 1.837098704611e+01 ||r(i)||/||b|| 7.937035135615e-02
   67 SNES Function norm 2.314589864333e+02 
      0 KSP preconditioned resid norm 4.021695169087e+04 true resid norm 2.314589864333e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.852569690755e+01 true resid norm 3.590909196842e+02 ||r(i)||/||b|| 1.551423538216e+00
      2 KSP preconditioned resid norm 1.348973307542e-04 true resid norm 6.035401149038e+01 ||r(i)||/||b|| 2.607546694143e-01
   68 SNES Function norm 2.314589801992e+02 
      0 KSP preconditioned resid norm 1.725952032453e+05 true resid norm 2.314589801992e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.777975072103e+01 true resid norm 3.518023660349e+02 ||r(i)||/||b|| 1.519933967272e+00
      2 KSP preconditioned resid norm 1.266007649822e-04 true resid norm 7.313840992175e+01 ||r(i)||/||b|| 3.159886467089e-01
   69 SNES Function norm 2.314589719235e+02 
      0 KSP preconditioned resid norm 3.607076901939e+04 true resid norm 2.314589719235e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.210132261593e+02 true resid norm 8.774444290589e+02 ||r(i)||/||b|| 3.790928568321e+00
      2 KSP preconditioned resid norm 4.891015819530e-04 true resid norm 8.098219226779e+01 ||r(i)||/||b|| 3.498770930970e-01
   70 SNES Function norm 2.314589643803e+02 
      0 KSP preconditioned resid norm 2.442776989201e+04 true resid norm 2.314589643803e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.014921648500e+02 true resid norm 2.188843891071e+03 ||r(i)||/||b|| 9.456725501782e+00
      2 KSP preconditioned resid norm 1.771597678165e-03 true resid norm 1.481761731272e+02 ||r(i)||/||b|| 6.401833410254e-01
   71 SNES Function norm 2.314589631504e+02 
      0 KSP preconditioned resid norm 2.803938255445e+04 true resid norm 2.314589631504e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.937985936300e+01 true resid norm 5.033022630111e+02 ||r(i)||/||b|| 2.174477307600e+00
      2 KSP preconditioned resid norm 3.691607880973e-04 true resid norm 2.715686837331e+01 ||r(i)||/||b|| 1.173290850511e-01
   72 SNES Function norm 2.314589440605e+02 
      0 KSP preconditioned resid norm 9.069257939432e+03 true resid norm 2.314589440605e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.034392926329e+01 true resid norm 7.627620750651e+01 ||r(i)||/||b|| 3.295453015052e-01
      2 KSP preconditioned resid norm 2.062351156668e-04 true resid norm 6.951980961959e+00 ||r(i)||/||b|| 3.003548206001e-02
   73 SNES Function norm 2.314585545983e+02 
      0 KSP preconditioned resid norm 1.678113228786e+04 true resid norm 2.314585545983e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.926568185998e+01 true resid norm 2.132342065400e+02 ||r(i)||/||b|| 9.212630179515e-01
      2 KSP preconditioned resid norm 7.165391508181e-05 true resid norm 1.872154768117e+01 ||r(i)||/||b|| 8.088509717718e-02
   74 SNES Function norm 2.314584904582e+02 
      0 KSP preconditioned resid norm 1.866490154072e+06 true resid norm 2.314584904582e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.383059477203e+01 true resid norm 2.450077515274e+02 ||r(i)||/||b|| 1.058538621946e+00
      2 KSP preconditioned resid norm 1.572755583893e-04 true resid norm 6.900003764147e+01 ||r(i)||/||b|| 2.981097712375e-01
   75 SNES Function norm 2.314584801547e+02 
      0 KSP preconditioned resid norm 6.156188967138e+04 true resid norm 2.314584801547e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.447913103987e+01 true resid norm 1.072790720471e+02 ||r(i)||/||b|| 4.634916464304e-01
      2 KSP preconditioned resid norm 5.644002952175e-05 true resid norm 1.929574883131e+01 ||r(i)||/||b|| 8.336591866678e-02
   76 SNES Function norm 2.314584104532e+02 
      0 KSP preconditioned resid norm 4.115894770006e+04 true resid norm 2.314584104532e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.929751893305e+01 true resid norm 3.567001584633e+02 ||r(i)||/||b|| 1.541098280960e+00
      2 KSP preconditioned resid norm 4.520979883364e-04 true resid norm 6.072093527410e+01 ||r(i)||/||b|| 2.623405870420e-01
   77 SNES Function norm 2.314584051144e+02 
      0 KSP preconditioned resid norm 2.095491764620e+05 true resid norm 2.314584051144e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.552674762551e+01 true resid norm 2.582375444470e+02 ||r(i)||/||b|| 1.115697415781e+00
      2 KSP preconditioned resid norm 7.435164108048e-05 true resid norm 1.988766346637e+01 ||r(i)||/||b|| 8.592327185759e-02
   78 SNES Function norm 2.314582888096e+02 
      0 KSP preconditioned resid norm 1.294622386441e+04 true resid norm 2.314582888096e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.107411018281e+01 true resid norm 8.058835795266e+01 ||r(i)||/||b|| 3.481765909838e-01
      2 KSP preconditioned resid norm 3.309588564016e-05 true resid norm 4.903075273077e+00 ||r(i)||/||b|| 2.118340759493e-02
   79 SNES Function norm 2.314571909137e+02 
      0 KSP preconditioned resid norm 1.204264280969e+04 true resid norm 2.314571909137e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.737477639497e+00 true resid norm 1.301274832418e+01 ||r(i)||/||b|| 5.622097232241e-02
      2 KSP preconditioned resid norm 1.116891328813e-05 true resid norm 6.616099462928e+00 ||r(i)||/||b|| 2.858454920674e-02
   80 SNES Function norm 2.314565930335e+02 
      0 KSP preconditioned resid norm 2.685765309163e+04 true resid norm 2.314565930335e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.392905224116e+01 true resid norm 1.017442509084e+02 ||r(i)||/||b|| 4.395824269896e-01
      2 KSP preconditioned resid norm 1.415949639751e-04 true resid norm 2.252094414009e+01 ||r(i)||/||b|| 9.730094029692e-02
   81 SNES Function norm 2.314565401448e+02 
      0 KSP preconditioned resid norm 4.814419294191e+05 true resid norm 2.314565401448e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 5.857262358676e+01 true resid norm 5.054713713870e+02 ||r(i)||/||b|| 2.183871629079e+00
      2 KSP preconditioned resid norm 1.243507216932e-03 true resid norm 3.772098143596e+02 ||r(i)||/||b|| 1.629721994996e+00
   82 SNES Function norm 2.314565398085e+02 
      0 KSP preconditioned resid norm 2.026301569876e+05 true resid norm 2.314565398085e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.987037388147e+01 true resid norm 2.882471210283e+02 ||r(i)||/||b|| 1.245361748114e+00
      2 KSP preconditioned resid norm 2.169665365314e-04 true resid norm 4.035341344938e+01 ||r(i)||/||b|| 1.743455314884e-01
   83 SNES Function norm 2.314565078487e+02 
      0 KSP preconditioned resid norm 3.094380381349e+04 true resid norm 2.314565078487e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.978293556536e+00 true resid norm 2.635581568290e+01 ||r(i)||/||b|| 1.138694086758e-01
      2 KSP preconditioned resid norm 1.890022323482e-05 true resid norm 1.597791541658e+01 ||r(i)||/||b|| 6.903204219699e-02
   84 SNES Function norm 2.314564079163e+02 
      0 KSP preconditioned resid norm 6.209269840680e+04 true resid norm 2.314564079163e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.995982106142e+01 true resid norm 1.468548855791e+02 ||r(i)||/||b|| 6.344818313788e-01
      2 KSP preconditioned resid norm 7.230034458440e-05 true resid norm 1.277927519154e+01 ||r(i)||/||b|| 5.521244931858e-02
   85 SNES Function norm 2.314562581879e+02 
      0 KSP preconditioned resid norm 2.316871639948e+04 true resid norm 2.314562581879e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.061899183336e+01 true resid norm 7.658836412307e+01 ||r(i)||/||b|| 3.308977891662e-01
      2 KSP preconditioned resid norm 1.593652837122e-04 true resid norm 8.955355396819e+00 ||r(i)||/||b|| 3.869135130297e-02
   86 SNES Function norm 2.314559345442e+02 
      0 KSP preconditioned resid norm 2.271158654394e+04 true resid norm 2.314559345442e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.441411557989e+01 true resid norm 1.074834857947e+02 ||r(i)||/||b|| 4.643799088859e-01
      2 KSP preconditioned resid norm 1.535735640673e-04 true resid norm 1.965949694313e+01 ||r(i)||/||b|| 8.493840083143e-02
   87 SNES Function norm 2.314559150625e+02 
      0 KSP preconditioned resid norm 3.119665022860e+04 true resid norm 2.314559150625e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.487979701070e+01 true resid norm 1.076952042274e+02 ||r(i)||/||b|| 4.652946726303e-01
      2 KSP preconditioned resid norm 2.534108740613e-04 true resid norm 2.577373574045e+01 ||r(i)||/||b|| 1.113548371986e-01
   88 SNES Function norm 2.314558771417e+02 
      0 KSP preconditioned resid norm 4.567599584574e+05 true resid norm 2.314558771417e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.380099279901e+00 true resid norm 1.760087395396e+02 ||r(i)||/||b|| 7.604418678547e-01
      2 KSP preconditioned resid norm 1.693278102818e-03 true resid norm 2.175019651802e+02 ||r(i)||/||b|| 9.397124318735e-01
   89 SNES Function norm 2.314558765490e+02 
      0 KSP preconditioned resid norm 6.893967531875e+05 true resid norm 2.314558765490e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.982785897195e+01 true resid norm 2.190225259947e+02 ||r(i)||/||b|| 9.462819836779e-01
      2 KSP preconditioned resid norm 2.006267159571e-04 true resid norm 2.365026583574e+01 ||r(i)||/||b|| 1.021804509281e-01
   90 SNES Function norm 2.314558316055e+02 
      0 KSP preconditioned resid norm 3.213811396790e+04 true resid norm 2.314558316055e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.863726299685e+01 true resid norm 1.346365978970e+02 ||r(i)||/||b|| 5.816945590139e-01
      2 KSP preconditioned resid norm 8.119950416130e-05 true resid norm 7.914584107016e+00 ||r(i)||/||b|| 3.419479238054e-02
   91 SNES Function norm 2.314553994731e+02 
      0 KSP preconditioned resid norm 1.428885884403e+04 true resid norm 2.314553994731e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 6.461923684043e+00 true resid norm 4.652208559908e+01 ||r(i)||/||b|| 2.009980570986e-01
      2 KSP preconditioned resid norm 6.018427586116e-05 true resid norm 6.443132757602e+00 ||r(i)||/||b|| 2.783747007963e-02
   92 SNES Function norm 2.314548022234e+02 
      0 KSP preconditioned resid norm 1.997882268842e+04 true resid norm 2.314548022234e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.990540367836e+01 true resid norm 1.444167499553e+02 ||r(i)||/||b|| 6.239522730484e-01
      2 KSP preconditioned resid norm 6.933227199420e-05 true resid norm 4.340884186507e+00 ||r(i)||/||b|| 1.875478125667e-02
   93 SNES Function norm 2.314535554897e+02 
      0 KSP preconditioned resid norm 5.317751765458e+03 true resid norm 2.314535554897e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.051772818836e+01 true resid norm 7.626983587464e+01 ||r(i)||/||b|| 3.295254450219e-01
      2 KSP preconditioned resid norm 1.062889530517e-04 true resid norm 4.148559709642e+00 ||r(i)||/||b|| 1.792394029491e-02
   94 SNES Function norm 2.314519884611e+02 
      0 KSP preconditioned resid norm 8.518590251100e+04 true resid norm 2.314519884611e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.400567581249e+01 true resid norm 5.350691354513e+02 ||r(i)||/||b|| 2.311793210371e+00
      2 KSP preconditioned resid norm 3.190984674306e-04 true resid norm 9.080604176445e+01 ||r(i)||/||b|| 3.923320873940e-01
   95 SNES Function norm 2.314519821638e+02 
      0 KSP preconditioned resid norm 3.971419728627e+04 true resid norm 2.314519821638e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 1.584583085310e+02 true resid norm 1.155052020747e+03 ||r(i)||/||b|| 4.990460699229e+00
      2 KSP preconditioned resid norm 8.131629823886e-04 true resid norm 1.353955847914e+02 ||r(i)||/||b|| 5.849834748684e-01
   96 SNES Function norm 2.314519808162e+02 
      0 KSP preconditioned resid norm 5.113274886640e+04 true resid norm 2.314519808162e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 8.658920186396e+01 true resid norm 6.338848251652e+02 ||r(i)||/||b|| 2.738731476524e+00
      2 KSP preconditioned resid norm 6.225448401968e-04 true resid norm 7.231481532235e+01 ||r(i)||/||b|| 3.124398204212e-01
   97 SNES Function norm 2.314519732830e+02 
      0 KSP preconditioned resid norm 2.374489458388e+04 true resid norm 2.314519732830e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 7.056237350775e+00 true resid norm 5.327919633484e+01 ||r(i)||/||b|| 2.301954724305e-01
      2 KSP preconditioned resid norm 8.341487773131e-05 true resid norm 1.097883645826e+01 ||r(i)||/||b|| 4.743462024773e-02
   98 SNES Function norm 2.314517417065e+02 
      0 KSP preconditioned resid norm 2.725936939754e+04 true resid norm 2.314517417065e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 4.470892119649e+00 true resid norm 3.482468106411e+01 ||r(i)||/||b|| 1.504619529209e-01
      2 KSP preconditioned resid norm 2.743947437756e-05 true resid norm 1.930558512719e+01 ||r(i)||/||b|| 8.341084402670e-02
   99 SNES Function norm 2.314516598045e+02 
      0 KSP preconditioned resid norm 1.323063699793e+05 true resid norm 2.314516598045e+02 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 2.183745290203e+03 true resid norm 1.606786653196e+04 ||r(i)||/||b|| 6.942212704601e+01
      2 KSP preconditioned resid norm 1.339289871939e-02 true resid norm 7.097265620768e+03 ||r(i)||/||b|| 3.066413793170e+01
  100 SNES Function norm 2.314516598037e+02 
  Nonlinear solve did not converge due to DIVERGED_MAX_IT iterations 100
-gideon

> On Sep 8, 2015, at 10:53 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Sep 8, 2015 at 9:48 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> Your comment about the different pieces scaling different made me think of something, and I made a slight modification.  It still fails, but the conditioning looks much better, no?
> 
> This is great. Now its clear that the nonlinear convergence is crappy.
> 
> Lets start with looking at the distribution of residuals. Take iterate 50,
> which is as converged as you get and save the residual (VecView with
> binary). Then we can load it up and play with it. I recommend
> 
>   a) Norms for both halves
> 
>   b) Plot of the big half
> 
> What usually happens is that some small part is screwing up, but you
> have to iterate the whole system, and finding the descent direction is
> hard. If you select the problem variables, Newton will converge, and then
> the big system will converge once the problem is removed. This is the classic
> "coarse basis selection" problem in the nonlinear regime.
> 
>    Matt
> 
>   Thanks,
> 
>     Matt
>  
> 
>       0 SNES Function norm 4.027474333769e-09 
>         0 KSP preconditioned resid norm 9.390682304142e-09 true resid norm 4.027474333769e-09 ||r(i)||/||b|| 1.000000000000e+00
>         1 KSP preconditioned resid norm 4.937459249116e-20 true resid norm 3.737772204700e-19 ||r(i)||/||b|| 9.280685350023e-11
>       1 SNES Function norm 2.666655509899e-12 
>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>     0 SNES Function norm 5.066222213175e+03 
>       0 KSP preconditioned resid norm 3.135087050015e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.220576053756e-11 true resid norm 1.206309587478e-07 ||r(i)||/||b|| 2.381083056997e-11
>     1 SNES Function norm 8.484697184819e+02 
>       0 KSP preconditioned resid norm 2.668083368240e+02 true resid norm 8.484697184819e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.388166746665e-10 true resid norm 2.735732864000e-08 ||r(i)||/||b|| 3.224314085004e-11
>     2 SNES Function norm 6.549559723821e+02 
>       0 KSP preconditioned resid norm 1.552154273063e+02 true resid norm 6.549559723821e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.740227244709e-10 true resid norm 1.800722435354e-08 ||r(i)||/||b|| 2.749379364852e-11
>     3 SNES Function norm 5.770723278635e+02 
>       0 KSP preconditioned resid norm 1.348596653013e+02 true resid norm 5.770723278635e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.142365551700e-10 true resid norm 7.387664544933e-09 ||r(i)||/||b|| 1.280197331985e-11
>     4 SNES Function norm 5.237702241066e+02 
>       0 KSP preconditioned resid norm 1.524401223470e+02 true resid norm 5.237702241066e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.168592748079e-10 true resid norm 9.512653260875e-09 ||r(i)||/||b|| 1.816188248788e-11
>     5 SNES Function norm 4.753909020302e+02 
>       0 KSP preconditioned resid norm 1.864261890160e+02 true resid norm 4.753909020302e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.744912816503e-10 true resid norm 1.889384889937e-08 ||r(i)||/||b|| 3.974381675939e-11
>     6 SNES Function norm 4.221784591168e+02 
>       0 KSP preconditioned resid norm 2.550992140633e+02 true resid norm 4.221784591168e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.697718851690e-10 true resid norm 1.666255394677e-08 ||r(i)||/||b|| 3.946803439861e-11
>     7 SNES Function norm 3.806525080909e+02 
>       0 KSP preconditioned resid norm 5.183605288372e+02 true resid norm 3.806525080909e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.240268561213e-09 true resid norm 1.041310587019e-07 ||r(i)||/||b|| 2.735593657958e-10
>     8 SNES Function norm 3.762054656363e+02 
>       0 KSP preconditioned resid norm 5.320776249886e+03 true resid norm 3.762054656363e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.328856456273e-09 true resid norm 7.860313076537e-08 ||r(i)||/||b|| 2.089367059897e-10
>     9 SNES Function norm 3.758975227207e+02 
>       0 KSP preconditioned resid norm 6.252711400154e+03 true resid norm 3.758975227207e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.900963654627e-09 true resid norm 1.347299589292e-07 ||r(i)||/||b|| 3.584220453330e-10
>    10 SNES Function norm 3.757032042008e+02 
>       0 KSP preconditioned resid norm 5.890497230519e+02 true resid norm 3.757032042008e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.497822807230e-09 true resid norm 3.605311757144e-08 ||r(i)||/||b|| 9.596169840535e-11
>    11 SNES Function norm 3.728798164183e+02 
>       0 KSP preconditioned resid norm 1.081356468099e+03 true resid norm 3.728798164183e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.713971332514e-09 true resid norm 3.705072372867e-08 ||r(i)||/||b|| 9.936371478768e-11
>    12 SNES Function norm 3.723078736486e+02 
>       0 KSP preconditioned resid norm 3.745123875883e+03 true resid norm 3.723078736486e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.885886422519e-08 true resid norm 2.837206039485e-07 ||r(i)||/||b|| 7.620591022374e-10
>    13 SNES Function norm 3.721848051981e+02 
>       0 KSP preconditioned resid norm 3.495480420016e+03 true resid norm 3.721848051981e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.874759607493e-08 true resid norm 2.751735241072e-07 ||r(i)||/||b|| 7.393464759014e-10
>    14 SNES Function norm 3.720227571463e+02 
>       0 KSP preconditioned resid norm 1.265998097646e+04 true resid norm 3.720227571463e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.125038941589e-08 true resid norm 5.141611324046e-07 ||r(i)||/||b|| 1.382069033488e-09
>    15 SNES Function norm 3.720051993666e+02 
>       0 KSP preconditioned resid norm 2.912369751490e+03 true resid norm 3.720051993666e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.170907138699e-09 true resid norm 3.623872061163e-08 ||r(i)||/||b|| 9.741455407971e-11
>    16 SNES Function norm 3.718945615434e+02 
>       0 KSP preconditioned resid norm 1.082053028195e+03 true resid norm 3.718945615434e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.660982704773e-09 true resid norm 9.903133228018e-08 ||r(i)||/||b|| 2.662887348210e-10
>    17 SNES Function norm 3.700412140239e+02 
>       0 KSP preconditioned resid norm 3.676306553630e+01 true resid norm 3.700412140239e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.389666775474e-10 true resid norm 4.384613797840e-10 ||r(i)||/||b|| 1.184898771183e-12
>    18 SNES Function norm 3.351961239413e+02 
>       0 KSP preconditioned resid norm 1.071519127966e+02 true resid norm 3.351961239413e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.391049839274e-11 true resid norm 1.771280721778e-09 ||r(i)||/||b|| 5.284311468017e-12
>    19 SNES Function norm 3.096040290055e+02 
>       0 KSP preconditioned resid norm 3.362840348158e+02 true resid norm 3.096040290055e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.847034649159e-10 true resid norm 3.697317355249e-09 ||r(i)||/||b|| 1.194208411023e-11
>    20 SNES Function norm 3.008687462984e+02 
>       0 KSP preconditioned resid norm 7.161404453567e+01 true resid norm 3.008687462984e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.352839847013e-11 true resid norm 7.837384020140e-10 ||r(i)||/||b|| 2.604917963917e-12
>    21 SNES Function norm 2.752608806757e+02 
>       0 KSP preconditioned resid norm 1.727563200247e+02 true resid norm 2.752608806757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.663026698374e-10 true resid norm 5.807801532466e-09 ||r(i)||/||b|| 2.109926233692e-11
>    22 SNES Function norm 2.707559177016e+02 
>       0 KSP preconditioned resid norm 4.125046548626e+02 true resid norm 2.707559177016e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.109007703208e-10 true resid norm 5.875389803064e-09 ||r(i)||/||b|| 2.169994972940e-11
>    23 SNES Function norm 2.698855074973e+02 
>       0 KSP preconditioned resid norm 7.414116220947e+01 true resid norm 2.698855074973e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.222534270039e-11 true resid norm 6.928234778488e-10 ||r(i)||/||b|| 2.567101450810e-12
>    24 SNES Function norm 2.616892369202e+02 
>       0 KSP preconditioned resid norm 1.819161466540e+02 true resid norm 2.616892369202e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.013662651542e-09 true resid norm 4.195842561304e-09 ||r(i)||/||b|| 1.603368411588e-11
>    25 SNES Function norm 2.600248325147e+02 
>       0 KSP preconditioned resid norm 1.265123744997e+02 true resid norm 2.600248325147e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.572744639399e-09 true resid norm 7.386966838391e-09 ||r(i)||/||b|| 2.840869761150e-11
>    26 SNES Function norm 2.593065715720e+02 
>       0 KSP preconditioned resid norm 1.314965938655e+02 true resid norm 2.593065715720e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.156889834113e-10 true resid norm 3.157778817609e-09 ||r(i)||/||b|| 1.217778168314e-11
>    27 SNES Function norm 2.517348459439e+02 
>       0 KSP preconditioned resid norm 2.618005944700e+01 true resid norm 2.517348459439e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.483924363849e-11 true resid norm 7.418968110702e-10 ||r(i)||/||b|| 2.947135936976e-12
>    28 SNES Function norm 2.298635352641e+02 
>       0 KSP preconditioned resid norm 8.770042104744e+01 true resid norm 2.298635352641e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.098440763657e-09 true resid norm 3.936819814733e-09 ||r(i)||/||b|| 1.712676962968e-11
>    29 SNES Function norm 2.269871118080e+02 
>       0 KSP preconditioned resid norm 2.285655807501e+02 true resid norm 2.269871118080e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.216116049541e-08 true resid norm 9.327420966977e-08 ||r(i)||/||b|| 4.109229327024e-10
>    30 SNES Function norm 2.252105956612e+02 
>       0 KSP preconditioned resid norm 8.597156834231e+01 true resid norm 2.252105956612e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.431615405953e-09 true resid norm 4.297181487549e-09 ||r(i)||/||b|| 1.908072519826e-11
>    31 SNES Function norm 2.251883212710e+02 
>       0 KSP preconditioned resid norm 7.066983917118e+01 true resid norm 2.251883212710e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.998573955346e-10 true resid norm 3.454505271056e-09 ||r(i)||/||b|| 1.534051700176e-11
>    32 SNES Function norm 2.117895362999e+02 
>       0 KSP preconditioned resid norm 3.830985324064e+02 true resid norm 2.117895362999e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.016573825110e-09 true resid norm 1.942657883557e-08 ||r(i)||/||b|| 9.172586698552e-11
>    33 SNES Function norm 2.116404150058e+02 
>       0 KSP preconditioned resid norm 5.329232125733e+02 true resid norm 2.116404150058e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.609537995048e-09 true resid norm 1.451505196923e-08 ||r(i)||/||b|| 6.858355465253e-11
>    34 SNES Function norm 2.115566314991e+02 
>       0 KSP preconditioned resid norm 6.609724236205e+02 true resid norm 2.115566314991e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.254105497370e-10 true resid norm 1.178528765891e-08 ||r(i)||/||b|| 5.570748397436e-11
>    35 SNES Function norm 2.115028944726e+02 
>       0 KSP preconditioned resid norm 8.217040556045e+02 true resid norm 2.115028944726e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.803324342385e-09 true resid norm 2.957366914143e-08 ||r(i)||/||b|| 1.398263093050e-10
>    36 SNES Function norm 2.114773510911e+02 
>       0 KSP preconditioned resid norm 1.091957913212e+03 true resid norm 2.114773510911e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.264170340401e-08 true resid norm 1.635833495717e-07 ||r(i)||/||b|| 7.735265678696e-10
>    37 SNES Function norm 2.114573155596e+02 
>       0 KSP preconditioned resid norm 1.295967660499e+03 true resid norm 2.114573155596e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.787403648651e-08 true resid norm 1.913077973453e-07 ||r(i)||/||b|| 9.047111793648e-10
>    38 SNES Function norm 2.114434864633e+02 
>       0 KSP preconditioned resid norm 1.567323979910e+03 true resid norm 2.114434864633e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.869044911030e-09 true resid norm 3.425540409553e-08 ||r(i)||/||b|| 1.620073744928e-10
>    39 SNES Function norm 2.114337906450e+02 
>       0 KSP preconditioned resid norm 1.835099219771e+03 true resid norm 2.114337906450e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.224665285731e-10 true resid norm 1.690151657152e-08 ||r(i)||/||b|| 7.993763210677e-11
>    40 SNES Function norm 2.114267348757e+02 
>       0 KSP preconditioned resid norm 2.146249211809e+03 true resid norm 2.114267348757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.809299780773e-08 true resid norm 1.157600089098e-07 ||r(i)||/||b|| 5.475183116166e-10
>    41 SNES Function norm 2.114215910347e+02 
>       0 KSP preconditioned resid norm 2.510692990698e+03 true resid norm 2.114215910347e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.485943641401e-08 true resid norm 9.439839396991e-08 ||r(i)||/||b|| 4.464936315535e-10
>    42 SNES Function norm 2.114206676373e+02 
>       0 KSP preconditioned resid norm 3.401544811763e+03 true resid norm 2.114206676373e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.910138749741e-09 true resid norm 3.806926432018e-08 ||r(i)||/||b|| 1.800640625423e-10
>    43 SNES Function norm 2.114186309697e+02 
>       0 KSP preconditioned resid norm 3.955962524189e+03 true resid norm 2.114186309697e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.500201400572e-08 true resid norm 1.570197988244e-07 ||r(i)||/||b|| 7.426961290224e-10
>    44 SNES Function norm 2.114171338084e+02 
>       0 KSP preconditioned resid norm 4.636713819556e+03 true resid norm 2.114171338084e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.248004040828e-09 true resid norm 7.296845022145e-08 ||r(i)||/||b|| 3.451397193171e-10
>    45 SNES Function norm 2.114170015592e+02 
>       0 KSP preconditioned resid norm 6.321621979244e+03 true resid norm 2.114170015592e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.674310076597e-08 true resid norm 1.788726595879e-07 ||r(i)||/||b|| 8.460656345928e-10
>    46 SNES Function norm 2.114164190412e+02 
>       0 KSP preconditioned resid norm 7.414534654352e+03 true resid norm 2.114164190412e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.520165386851e-08 true resid norm 4.326674448851e-07 ||r(i)||/||b|| 2.046517705897e-09
>    47 SNES Function norm 2.114162562951e+02 
>       0 KSP preconditioned resid norm 9.810108530473e+03 true resid norm 2.114162562951e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.690981568660e-08 true resid norm 1.826808079437e-07 ||r(i)||/||b|| 8.640811787375e-10
>    48 SNES Function norm 2.114160137794e+02 
>       0 KSP preconditioned resid norm 1.136099973228e+04 true resid norm 2.114160137794e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.015775310105e-08 true resid norm 1.261980904678e-07 ||r(i)||/||b|| 5.969183138583e-10
>    49 SNES Function norm 2.114158331092e+02 
>       0 KSP preconditioned resid norm 1.315560026827e+04 true resid norm 2.114158331092e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.131165057283e-08 true resid norm 3.097361970470e-07 ||r(i)||/||b|| 1.465056767470e-09
>    50 SNES Function norm 2.114158223286e+02 
>       0 KSP preconditioned resid norm 1.794717411653e+04 true resid norm 2.114158223286e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.070352092082e-07 true resid norm 6.224398071454e-07 ||r(i)||/||b|| 2.944149592446e-09
>    51 SNES Function norm 2.114157500757e+02 
>       0 KSP preconditioned resid norm 2.077631649185e+04 true resid norm 2.114157500757e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.358541701826e-08 true resid norm 5.604873460299e-07 ||r(i)||/||b|| 2.651114431301e-09
>    52 SNES Function norm 2.114156961871e+02 
>       0 KSP preconditioned resid norm 2.405048247398e+04 true resid norm 2.114156961871e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.439555573169e-08 true resid norm 3.896791261159e-07 ||r(i)||/||b|| 1.843189191455e-09
>    53 SNES Function norm 2.114156592651e+02 
>       0 KSP preconditioned resid norm 2.914886710376e+04 true resid norm 2.114156592651e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.436687722935e-07 true resid norm 1.370484810325e-06 ||r(i)||/||b|| 6.482418639609e-09
>    54 SNES Function norm 2.114156319126e+02 
>       0 KSP preconditioned resid norm 3.373835363786e+04 true resid norm 2.114156319126e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.028533154395e-07 true resid norm 6.668879459515e-07 ||r(i)||/||b|| 3.154392794508e-09
>    55 SNES Function norm 2.114156115019e+02 
>       0 KSP preconditioned resid norm 3.904963797200e+04 true resid norm 2.114156115019e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.986152192528e-08 true resid norm 4.508900491313e-07 ||r(i)||/||b|| 2.132718799374e-09
>    56 SNES Function norm 2.114155964342e+02 
>       0 KSP preconditioned resid norm 4.594596937760e+04 true resid norm 2.114155964342e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.567731897537e-08 true resid norm 5.054454305445e-07 ||r(i)||/||b|| 2.390766996709e-09
>    57 SNES Function norm 2.114155854345e+02 
>       0 KSP preconditioned resid norm 5.317614782539e+04 true resid norm 2.114155854345e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.460625544912e-07 true resid norm 9.171801951021e-07 ||r(i)||/||b|| 4.338280894556e-09
>    58 SNES Function norm 2.114155777135e+02 
>       0 KSP preconditioned resid norm 6.400017583847e+04 true resid norm 2.114155777135e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.093685259962e-08 true resid norm 6.506435082837e-07 ||r(i)||/||b|| 3.077557081273e-09
>    59 SNES Function norm 2.114155720468e+02 
>       0 KSP preconditioned resid norm 7.406807436961e+04 true resid norm 2.114155720468e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.571128588743e-07 true resid norm 1.516640238741e-06 ||r(i)||/||b|| 7.173739493538e-09
>    60 SNES Function norm 2.114155680388e+02 
>       0 KSP preconditioned resid norm 8.892466432276e+04 true resid norm 2.114155680388e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.335222631846e-08 true resid norm 8.161361513986e-07 ||r(i)||/||b|| 3.860340839464e-09
>    61 SNES Function norm 2.114155651044e+02 
>       0 KSP preconditioned resid norm 1.029106627628e+05 true resid norm 2.114155651044e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.244108037270e-07 true resid norm 3.009616926997e-06 ||r(i)||/||b|| 1.423555037450e-08
>    62 SNES Function norm 2.114155629136e+02 
>       0 KSP preconditioned resid norm 1.190953464375e+05 true resid norm 2.114155629136e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.108162336627e-08 true resid norm 9.193387836453e-07 ||r(i)||/||b|| 4.348491525295e-09
>    63 SNES Function norm 2.114155612779e+02 
>       0 KSP preconditioned resid norm 1.378243866677e+05 true resid norm 2.114155612779e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.138603793608e-07 true resid norm 2.482322226118e-06 ||r(i)||/||b|| 1.174143573497e-08
>    64 SNES Function norm 2.114155602404e+02 
>       0 KSP preconditioned resid norm 1.698559088929e+05 true resid norm 2.114155602404e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.608494949489e-08 true resid norm 1.125534374356e-06 ||r(i)||/||b|| 5.323801015765e-09
>    65 SNES Function norm 2.114155594364e+02 
>       0 KSP preconditioned resid norm 1.965648734846e+05 true resid norm 2.114155594364e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.689853079655e-07 true resid norm 2.539748843840e-06 ||r(i)||/||b|| 1.201306493529e-08
>    66 SNES Function norm 2.114155588361e+02 
>       0 KSP preconditioned resid norm 2.274726625349e+05 true resid norm 2.114155588361e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.937782777025e-07 true resid norm 5.126870538363e-06 ||r(i)||/||b|| 2.425020450996e-08
>    67 SNES Function norm 2.114155583879e+02 
>       0 KSP preconditioned resid norm 2.632394448333e+05 true resid norm 2.114155583879e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.229084794159e-07 true resid norm 3.026195587359e-06 ||r(i)||/||b|| 1.431396823599e-08
>    68 SNES Function norm 2.114155581765e+02 
>       0 KSP preconditioned resid norm 3.367643572924e+05 true resid norm 2.114155581765e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.632794900620e-08 true resid norm 2.446819364813e-06 ||r(i)||/||b|| 1.157350663271e-08
>    69 SNES Function norm 2.114155579720e+02 
>       0 KSP preconditioned resid norm 3.897127303190e+05 true resid norm 2.114155579720e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.933486330809e-06 true resid norm 2.730183810468e-05 ||r(i)||/||b|| 1.291382638372e-07
>    70 SNES Function norm 2.114155578193e+02 
>       0 KSP preconditioned resid norm 4.509848215072e+05 true resid norm 2.114155578193e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.706389710731e-08 true resid norm 3.785217910694e-06 ||r(i)||/||b|| 1.790415970205e-08
>    71 SNES Function norm 2.114155577054e+02 
>       0 KSP preconditioned resid norm 5.242625554614e+05 true resid norm 2.114155577054e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.321749665453e-06 true resid norm 9.069981074375e-06 ||r(i)||/||b|| 4.290119976419e-08
>    72 SNES Function norm 2.114155576210e+02 
>       0 KSP preconditioned resid norm 6.066871925199e+05 true resid norm 2.114155576210e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.327204488687e-07 true resid norm 7.090997425424e-06 ||r(i)||/||b|| 3.354056581842e-08
>    73 SNES Function norm 2.114155575580e+02 
>       0 KSP preconditioned resid norm 7.020695777561e+05 true resid norm 2.114155575580e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.123816847857e-06 true resid norm 8.125899637337e-06 ||r(i)||/||b|| 3.843567489166e-08
>    74 SNES Function norm 2.114155575109e+02 
>       0 KSP preconditioned resid norm 8.124474305522e+05 true resid norm 2.114155575109e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.293550435673e-07 true resid norm 7.844829319667e-06 ||r(i)||/||b|| 3.710620643072e-08
>    75 SNES Function norm 2.114155574865e+02 
>       0 KSP preconditioned resid norm 1.029314742970e+06 true resid norm 2.114155574865e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.033392384072e-07 true resid norm 9.399925240813e-06 ||r(i)||/||b|| 4.446184260312e-08
>    76 SNES Function norm 2.114155574646e+02 
>       0 KSP preconditioned resid norm 1.191137352032e+06 true resid norm 2.114155574646e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.636526184597e-07 true resid norm 1.077765913224e-05 ||r(i)||/||b|| 5.097855267366e-08
>    77 SNES Function norm 2.114155574575e+02 
>       0 KSP preconditioned resid norm 1.563965951801e+06 true resid norm 2.114155574575e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.137216998536e-06 true resid norm 5.108848518118e-05 ||r(i)||/||b|| 2.416496013612e-07
>    78 SNES Function norm 2.114155574558e+02 
>       0 KSP preconditioned resid norm 2.108895892585e+06 true resid norm 2.114155574558e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.138935971807e-05 true resid norm 6.467637231732e-05 ||r(i)||/||b|| 3.059205911601e-07
>    79 SNES Function norm 2.114155574511e+02 
>       0 KSP preconditioned resid norm 2.564542138115e+06 true resid norm 2.114155574511e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.391129540980e-06 true resid norm 4.283623913226e-05 ||r(i)||/||b|| 2.026163052933e-07
>    80 SNES Function norm 2.114155574475e+02 
>       0 KSP preconditioned resid norm 2.967732587697e+06 true resid norm 2.114155574475e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.446508619620e-06 true resid norm 5.164837794317e-05 ||r(i)||/||b|| 2.442979058246e-07
>    81 SNES Function norm 2.114155574449e+02 
>       0 KSP preconditioned resid norm 3.434290246612e+06 true resid norm 2.114155574449e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.781979196562e-06 true resid norm 4.886329177979e-05 ||r(i)||/||b|| 2.311243901364e-07
>    82 SNES Function norm 2.114155574429e+02 
>       0 KSP preconditioned resid norm 3.974205574181e+06 true resid norm 2.114155574429e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.425678262963e-05 true resid norm 8.517034218173e-05 ||r(i)||/||b|| 4.028574964485e-07
>    83 SNES Function norm 2.114155574416e+02 
>       0 KSP preconditioned resid norm 4.785116089506e+06 true resid norm 2.114155574416e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.130667094145e-05 true resid norm 1.264608296603e-04 ||r(i)||/||b|| 5.981623641639e-07
>    84 SNES Function norm 2.114155574406e+02 
>       0 KSP preconditioned resid norm 5.537350742612e+06 true resid norm 2.114155574406e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.299149833543e-06 true resid norm 5.419308321331e-05 ||r(i)||/||b|| 2.563344148812e-07
>    85 SNES Function norm 2.114155574399e+02 
>       0 KSP preconditioned resid norm 6.767161611151e+06 true resid norm 2.114155574399e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.781986133631e-05 true resid norm 1.232616969009e-04 ||r(i)||/||b|| 5.830303994346e-07
>    86 SNES Function norm 2.114155574394e+02 
>       0 KSP preconditioned resid norm 7.867884017146e+06 true resid norm 2.114155574394e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.002696664504e-05 true resid norm 1.398533205346e-04 ||r(i)||/||b|| 6.615091255747e-07
>    87 SNES Function norm 2.114155574390e+02 
>       0 KSP preconditioned resid norm 9.102289060164e+06 true resid norm 2.114155574390e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.231201322143e-05 true resid norm 1.368407286352e-04 ||r(i)||/||b|| 6.472595030036e-07
>    88 SNES Function norm 2.114155574387e+02 
>       0 KSP preconditioned resid norm 1.063720954841e+07 true resid norm 2.114155574387e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.517853677131e-05 true resid norm 1.273061147947e-04 ||r(i)||/||b|| 6.021605805032e-07
>    89 SNES Function norm 2.114155574385e+02 
>       0 KSP preconditioned resid norm 1.230866787650e+07 true resid norm 2.114155574385e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.745229646688e-05 true resid norm 1.389443348166e-04 ||r(i)||/||b|| 6.572096041560e-07
>    90 SNES Function norm 2.114155574384e+02 
>       0 KSP preconditioned resid norm 1.436772950622e+07 true resid norm 2.114155574384e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.955301916175e-05 true resid norm 5.633754040143e-04 ||r(i)||/||b|| 2.664777421494e-06
>    91 SNES Function norm 2.114155574383e+02 
>       0 KSP preconditioned resid norm 1.662010558966e+07 true resid norm 2.114155574383e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.147116976734e-05 true resid norm 1.351152564949e-04 ||r(i)||/||b|| 6.390979837628e-07
>    92 SNES Function norm 2.114155574382e+02 
>       0 KSP preconditioned resid norm 1.923255924422e+07 true resid norm 2.114155574382e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.030030646485e-05 true resid norm 1.877965090103e-04 ||r(i)||/||b|| 8.882814078868e-07
>    93 SNES Function norm 2.114155574381e+02 
>       0 KSP preconditioned resid norm 2.226793244995e+07 true resid norm 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.411985876631e-04 true resid norm 1.342893408929e-03 ||r(i)||/||b|| 6.351913857247e-06
>    94 SNES Function norm 2.114155574381e+02 
>       0 KSP preconditioned resid norm 2.576953439486e+07 true resid norm 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.147595096922e-05 true resid norm 4.578229507611e-04 ||r(i)||/||b|| 2.165512114193e-06
>    95 SNES Function norm 2.114155574380e+02 
>       0 KSP preconditioned resid norm 2.981662207354e+07 true resid norm 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.456246888386e-06 true resid norm 2.856742906943e-04 ||r(i)||/||b|| 1.351245358460e-06
>    96 SNES Function norm 2.114155574380e+02 
>       0 KSP preconditioned resid norm 3.451395888978e+07 true resid norm 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.972976681165e-05 true resid norm 3.126288732572e-04 ||r(i)||/||b|| 1.478741096662e-06
>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 96
> 
> -gideon
> 
>> On Sep 8, 2015, at 10:31 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>> 
>> On Tue, Sep 8, 2015 at 9:19 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>> Ok, going back and running the problem with the flags, -snes_fd -snes_mf_operator -ksp_monitor_true_residual -snes_max_funcs 500000, during the grid sequence, I still get:
>> 
>> Lets try using -pc_type lu instead of MUMPS, just to make sure. If we still
>> get this, then the problem is really horribly conditioned without being singular.
>> We are losing all precision for some reason. Is it possible that the two halves
>> are so differently scaled that we make no progress on one with LU?
>> 
>>   Thanks,
>> 
>>      Matt
>>  
>>       0 SNES Function norm 9.131726454160e-09 
>>         0 KSP preconditioned resid norm 4.201012306235e-09 true resid norm 9.131726454160e-09 ||r(i)||/||b|| 1.000000000000e+00
>>         1 KSP preconditioned resid norm 4.811278613811e-14 true resid norm 7.117495013042e-13 ||r(i)||/||b|| 7.794249037978e-05
>>         2 KSP preconditioned resid norm 1.729322442321e-19 true resid norm 2.611605096203e-13 ||r(i)||/||b|| 2.859924800982e-05
>>       1 SNES Function norm 2.066497600915e-12 
>>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>     0 SNES Function norm 5.066222213176e+03 
>>       0 KSP preconditioned resid norm 3.135051257976e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.425321207844e-04 true resid norm 1.654961474093e+00 ||r(i)||/||b|| 3.266657885218e-04
>>       2 KSP preconditioned resid norm 2.739130995469e-09 true resid norm 7.943827483388e-03 ||r(i)||/||b|| 1.567998234015e-06
>>     1 SNES Function norm 8.487402654330e+02 
>>       0 KSP preconditioned resid norm 2.670376525819e+02 true resid norm 8.487402654330e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.518628784729e-02 true resid norm 1.820466086409e+00 ||r(i)||/||b|| 2.144903641964e-03
>>       2 KSP preconditioned resid norm 1.680654269556e-07 true resid norm 1.635299027240e-01 ||r(i)||/||b|| 1.926736710678e-04
>>     2 SNES Function norm 6.547660268557e+02 
>>       0 KSP preconditioned resid norm 1.529984375428e+02 true resid norm 6.547660268557e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.106652647444e-02 true resid norm 9.350438798847e-01 ||r(i)||/||b|| 1.428058026124e-03
>>       2 KSP preconditioned resid norm 1.739404692247e-07 true resid norm 6.642022841611e-02 ||r(i)||/||b|| 1.014411647701e-04
>>     3 SNES Function norm 5.771035131885e+02 
>>       0 KSP preconditioned resid norm 1.360697738776e+02 true resid norm 5.771035131885e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.323138791370e-03 true resid norm 6.419077863137e-02 ||r(i)||/||b|| 1.112292286642e-04
>>     4 SNES Function norm 5.238679723156e+02 
>>       0 KSP preconditioned resid norm 1.528156345475e+02 true resid norm 5.238679723156e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.115694692015e-02 true resid norm 1.230231531838e+00 ||r(i)||/||b|| 2.348361795054e-03
>>       2 KSP preconditioned resid norm 5.444815701049e-08 true resid norm 6.640979326953e-02 ||r(i)||/||b|| 1.267681873660e-04
>>     5 SNES Function norm 4.755828064476e+02 
>>       0 KSP preconditioned resid norm 1.872910183528e+02 true resid norm 4.755828064476e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.182180755863e-02 true resid norm 5.918459188490e-01 ||r(i)||/||b|| 1.244464498769e-03
>>       2 KSP preconditioned resid norm 7.426717097996e-08 true resid norm 8.473303784961e-02 ||r(i)||/||b|| 1.781667392111e-04
>>     6 SNES Function norm 4.224257767287e+02 
>>       0 KSP preconditioned resid norm 2.578229460575e+02 true resid norm 4.224257767287e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.258348180527e-02 true resid norm 3.257118054248e-01 ||r(i)||/||b|| 7.710509712432e-04
>>       2 KSP preconditioned resid norm 1.122972268815e-07 true resid norm 1.274729638407e-01 ||r(i)||/||b|| 3.017641698568e-04
>>     7 SNES Function norm 3.811223987907e+02 
>>       0 KSP preconditioned resid norm 5.218684581634e+02 true resid norm 3.811223987907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.089005298384e-01 true resid norm 5.194667956652e+00 ||r(i)||/||b|| 1.362992039601e-02
>>       2 KSP preconditioned resid norm 7.570453480881e-07 true resid norm 2.863356359719e-01 ||r(i)||/||b|| 7.512957435207e-04
>>     8 SNES Function norm 3.771290745624e+02 
>>       0 KSP preconditioned resid norm 5.827455360446e+03 true resid norm 3.771290745624e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.620225652579e-01 true resid norm 1.332007773441e+01 ||r(i)||/||b|| 3.531967867995e-02
>>       2 KSP preconditioned resid norm 3.596801636560e-06 true resid norm 3.425465452374e+00 ||r(i)||/||b|| 9.083005483862e-03
>>     9 SNES Function norm 3.767897776540e+02 
>>       0 KSP preconditioned resid norm 5.185115189077e+03 true resid norm 3.767897776540e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.021105455600e-01 true resid norm 1.043813038739e+01 ||r(i)||/||b|| 2.770279611187e-02
>>       2 KSP preconditioned resid norm 6.951879108628e-06 true resid norm 2.816562934830e+00 ||r(i)||/||b|| 7.475157506572e-03
>>    10 SNES Function norm 3.746968158051e+02 
>>       0 KSP preconditioned resid norm 8.165048723911e+02 true resid norm 3.746968158051e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.177528145230e-01 true resid norm 2.394044923481e+00 ||r(i)||/||b|| 6.389285476945e-03
>>       2 KSP preconditioned resid norm 7.325770200602e-07 true resid norm 4.877299135281e-01 ||r(i)||/||b|| 1.301665487816e-03
>>    11 SNES Function norm 3.726877791108e+02 
>>       0 KSP preconditioned resid norm 5.024547551936e+02 true resid norm 3.726877791108e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.640017871568e-01 true resid norm 1.648101091752e+00 ||r(i)||/||b|| 4.422203206351e-03
>>       2 KSP preconditioned resid norm 1.005738633834e-06 true resid norm 2.935508517682e-01 ||r(i)||/||b|| 7.876589151074e-04
>>    12 SNES Function norm 3.668832918639e+02 
>>       0 KSP preconditioned resid norm 7.762104953873e+02 true resid norm 3.668832918639e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.912259329681e-02 true resid norm 1.334600681831e+00 ||r(i)||/||b|| 3.637670919956e-03
>>       2 KSP preconditioned resid norm 7.640320635302e-07 true resid norm 4.471334143387e-01 ||r(i)||/||b|| 1.218734742776e-03
>>    13 SNES Function norm 3.631997889667e+02 
>>       0 KSP preconditioned resid norm 2.515761606628e+02 true resid norm 3.631997889667e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.648297544435e-01 true resid norm 2.866637955219e+00 ||r(i)||/||b|| 7.892730233611e-03
>>       2 KSP preconditioned resid norm 7.305731711999e-07 true resid norm 1.472589913546e-01 ||r(i)||/||b|| 4.054490003243e-04
>>    14 SNES Function norm 3.557844076846e+02 
>>       0 KSP preconditioned resid norm 6.995145168850e+02 true resid norm 3.557844076846e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.626527197886e-02 true resid norm 5.642316830909e-01 ||r(i)||/||b|| 1.585880861848e-03
>>       2 KSP preconditioned resid norm 1.040702263139e-06 true resid norm 4.073703976052e-01 ||r(i)||/||b|| 1.144992272866e-03
>>    15 SNES Function norm 3.534147071188e+02 
>>       0 KSP preconditioned resid norm 2.698507093385e+02 true resid norm 3.534147071188e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.592052963868e-01 true resid norm 1.332346399928e+00 ||r(i)||/||b|| 3.769923472595e-03
>>       2 KSP preconditioned resid norm 5.021635239169e-07 true resid norm 1.573071613558e-01 ||r(i)||/||b|| 4.451064378114e-04
>>    16 SNES Function norm 3.499900270010e+02 
>>       0 KSP preconditioned resid norm 7.295084253442e+01 true resid norm 3.499900270010e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.040248007780e-02 true resid norm 1.427506717563e-01 ||r(i)||/||b|| 4.078706841436e-04
>>       2 KSP preconditioned resid norm 1.246491224109e-07 true resid norm 4.074099442269e-02 ||r(i)||/||b|| 1.164061581177e-04
>>    17 SNES Function norm 3.473017914657e+02 
>>       0 KSP preconditioned resid norm 4.169372990718e+02 true resid norm 3.473017914657e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.140552389441e-01 true resid norm 1.893314948416e+00 ||r(i)||/||b|| 5.451497789361e-03
>>       2 KSP preconditioned resid norm 7.751532205539e-07 true resid norm 2.494992549211e-01 ||r(i)||/||b|| 7.183932275966e-04
>>    18 SNES Function norm 3.464416771335e+02 
>>       0 KSP preconditioned resid norm 2.968698304201e+02 true resid norm 3.464416771335e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.683113115222e-02 true resid norm 2.501212385243e-01 ||r(i)||/||b|| 7.219721385540e-04
>>       2 KSP preconditioned resid norm 5.274081741342e-07 true resid norm 1.712806350336e-01 ||r(i)||/||b|| 4.943996243489e-04
>>    19 SNES Function norm 3.460747441983e+02 
>>       0 KSP preconditioned resid norm 3.707198448214e+02 true resid norm 3.460747441983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.740664131652e-02 true resid norm 9.477346570578e-01 ||r(i)||/||b|| 2.738525919461e-03
>>       2 KSP preconditioned resid norm 4.649501179917e-07 true resid norm 2.217443031583e-01 ||r(i)||/||b|| 6.407410736429e-04
>>    20 SNES Function norm 3.455596398964e+02 
>>       0 KSP preconditioned resid norm 5.000331867417e+02 true resid norm 3.455596398964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.821124737573e-02 true resid norm 7.651194558377e-01 ||r(i)||/||b|| 2.214145888296e-03
>>       2 KSP preconditioned resid norm 5.030283955029e-07 true resid norm 2.918161448112e-01 ||r(i)||/||b|| 8.444740389784e-04
>>    21 SNES Function norm 3.453222949611e+02 
>>       0 KSP preconditioned resid norm 7.399233760260e+02 true resid norm 3.453222949611e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.030913689264e-01 true resid norm 8.930798988128e-01 ||r(i)||/||b|| 2.586221370136e-03
>>       2 KSP preconditioned resid norm 1.497261432173e-06 true resid norm 4.385255355661e-01 ||r(i)||/||b|| 1.269902181136e-03
>>    22 SNES Function norm 3.452850731057e+02 
>>       0 KSP preconditioned resid norm 1.850020325701e+03 true resid norm 3.452850731057e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.670597549481e-01 true resid norm 6.205758169419e+00 ||r(i)||/||b|| 1.797285389027e-02
>>       2 KSP preconditioned resid norm 4.122440586032e-06 true resid norm 1.109676174925e+00 ||r(i)||/||b|| 3.213797124053e-03
>>    23 SNES Function norm 3.452722963704e+02 
>>       0 KSP preconditioned resid norm 3.362446711623e+03 true resid norm 3.452722963704e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.181412468129e+00 true resid norm 1.564006484993e+01 ||r(i)||/||b|| 4.529776936738e-02
>>       2 KSP preconditioned resid norm 1.165193864469e-05 true resid norm 1.869411988347e+00 ||r(i)||/||b|| 5.414312147250e-03
>>    24 SNES Function norm 3.452679622612e+02 
>>       0 KSP preconditioned resid norm 6.149140486077e+03 true resid norm 3.452679622612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.400761945257e-01 true resid norm 3.795886928810e+00 ||r(i)||/||b|| 1.099403171945e-02
>>       2 KSP preconditioned resid norm 1.754461070240e-06 true resid norm 3.648059240080e+00 ||r(i)||/||b|| 1.056587821294e-02
>>    25 SNES Function norm 3.452667776779e+02 
>>       0 KSP preconditioned resid norm 1.101683177789e+04 true resid norm 3.452667776779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.317993935110e+00 true resid norm 2.454711939414e+01 ||r(i)||/||b|| 7.109609432809e-02
>>       2 KSP preconditioned resid norm 3.457477291813e-05 true resid norm 6.807469339003e+00 ||r(i)||/||b|| 1.971654899665e-02
>>    26 SNES Function norm 3.452666586474e+02 
>>       0 KSP preconditioned resid norm 3.879190850784e+04 true resid norm 3.452666586474e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.739562350939e+01 true resid norm 1.221587956319e+02 ||r(i)||/||b|| 3.538099974971e-01
>>       2 KSP preconditioned resid norm 7.455463420492e-05 true resid norm 3.080916578781e+01 ||r(i)||/||b|| 8.923295955800e-02
>>    27 SNES Function norm 3.452666425264e+02 
>>       0 KSP preconditioned resid norm 5.709318799171e+04 true resid norm 3.452666425264e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.292899448408e-01 true resid norm 2.801354737138e+01 ||r(i)||/||b|| 8.113597990933e-02
>>    28 SNES Function norm 3.452666374187e+02 
>>       0 KSP preconditioned resid norm 2.488913348294e+05 true resid norm 3.452666374187e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.691998534170e+02 true resid norm 7.244072115265e+03 ||r(i)||/||b|| 2.098109498625e+01
>>       2 KSP preconditioned resid norm 2.503929513070e-03 true resid norm 2.928791518636e+03 ||r(i)||/||b|| 8.482694825461e+00
>>    29 SNES Function norm 3.452666374177e+02 
>>       0 KSP preconditioned resid norm 2.533150641403e+05 true resid norm 3.452666374177e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.817060595281e+02 true resid norm 1.333479168158e+03 ||r(i)||/||b|| 3.862172082803e+00
>>       2 KSP preconditioned resid norm 8.930133492283e-04 true resid norm 5.412764197416e+02 ||r(i)||/||b|| 1.567705538507e+00
>>    30 SNES Function norm 3.452666373128e+02 
>>       0 KSP preconditioned resid norm 2.169409532487e+05 true resid norm 3.452666373128e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.325438416139e+02 true resid norm 9.441618139691e+02 ||r(i)||/||b|| 2.734587451940e+00
>>       2 KSP preconditioned resid norm 8.796591247860e-04 true resid norm 2.978002325996e+02 ||r(i)||/||b|| 8.625224693509e-01
>>    31 SNES Function norm 3.452666372449e+02 
>>       0 KSP preconditioned resid norm 1.459475274582e+05 true resid norm 3.452666372449e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.978987062939e+02 true resid norm 1.411576222768e+03 ||r(i)||/||b|| 4.088365542735e+00
>>       2 KSP preconditioned resid norm 1.235854958341e-03 true resid norm 3.579390387115e+02 ||r(i)||/||b|| 1.036703231936e+00
>>    32 SNES Function norm 3.452666369928e+02 
>>       0 KSP preconditioned resid norm 1.264234515603e+05 true resid norm 3.452666369928e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.676975931928e+01 true resid norm 1.855746501202e+02 ||r(i)||/||b|| 5.374821376792e-01
>>       2 KSP preconditioned resid norm 9.286140031498e-05 true resid norm 3.495715551553e+01 ||r(i)||/||b|| 1.012468387331e-01
>>    33 SNES Function norm 3.452666245901e+02 
>>       0 KSP preconditioned resid norm 1.544591438108e+06 true resid norm 3.452666245901e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.501087626485e+01 true resid norm 3.233291920369e+02 ||r(i)||/||b|| 9.364623424597e-01
>>       2 KSP preconditioned resid norm 1.867215855867e-04 true resid norm 7.114686634655e+01 ||r(i)||/||b|| 2.060635499623e-01
>>    34 SNES Function norm 3.452666216061e+02 
>>       0 KSP preconditioned resid norm 3.431570281248e+05 true resid norm 3.452666216061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.140176766342e+02 true resid norm 1.760366951993e+03 ||r(i)||/||b|| 5.098572644538e+00
>>       2 KSP preconditioned resid norm 2.324744486242e-03 true resid norm 1.080605579953e+03 ||r(i)||/||b|| 3.129771348664e+00
>>    35 SNES Function norm 3.452666215921e+02 
>>       0 KSP preconditioned resid norm 3.734063885343e+05 true resid norm 3.452666215921e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.279219133674e+00 true resid norm 1.691481367261e+02 ||r(i)||/||b|| 4.899058470990e-01
>>       2 KSP preconditioned resid norm 4.046466703016e-04 true resid norm 1.964786369469e+02 ||r(i)||/||b|| 5.690635139908e-01
>>    36 SNES Function norm 3.452666212200e+02 
>>       0 KSP preconditioned resid norm 7.952390829283e+05 true resid norm 3.452666212200e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.707607129733e+01 true resid norm 2.806640653978e+02 ||r(i)||/||b|| 8.128908158167e-01
>>       2 KSP preconditioned resid norm 1.296391885225e-04 true resid norm 1.079623998255e+02 ||r(i)||/||b|| 3.126928384910e-01
>>    37 SNES Function norm 3.452666199636e+02 
>>       0 KSP preconditioned resid norm 9.358626316710e+05 true resid norm 3.452666199636e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.722696571742e+01 true resid norm 3.251492717238e+02 ||r(i)||/||b|| 9.417338744130e-01
>>       2 KSP preconditioned resid norm 2.862108374145e-04 true resid norm 3.320104402606e+01 ||r(i)||/||b|| 9.616059620694e-02
>>    38 SNES Function norm 3.452666133867e+02 
>>       0 KSP preconditioned resid norm 8.365871848022e+04 true resid norm 3.452666133867e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.482543031390e+02 true resid norm 1.772891732407e+03 ||r(i)||/||b|| 5.134848443690e+00
>>       2 KSP preconditioned resid norm 1.439398367387e-03 true resid norm 3.088902452162e+02 ||r(i)||/||b|| 8.946426710257e-01
>>    39 SNES Function norm 3.452666132332e+02 
>>       0 KSP preconditioned resid norm 8.960112658041e+04 true resid norm 3.452666132332e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.505891399150e+01 true resid norm 1.787713293837e+02 ||r(i)||/||b|| 5.177776319280e-01
>>       2 KSP preconditioned resid norm 6.384856870578e-05 true resid norm 2.543851208517e+01 ||r(i)||/||b|| 7.367787996341e-02
>>    40 SNES Function norm 3.452665904125e+02 
>>       0 KSP preconditioned resid norm 7.796808144307e+05 true resid norm 3.452665904125e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.020764189760e+01 true resid norm 6.961545395677e+02 ||r(i)||/||b|| 2.016281212544e+00
>>       2 KSP preconditioned resid norm 3.506849883654e-04 true resid norm 4.956460306859e+02 ||r(i)||/||b|| 1.435545878024e+00
>>    41 SNES Function norm 3.452665902941e+02 
>>       0 KSP preconditioned resid norm 4.990797566798e+05 true resid norm 3.452665902941e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.012324354359e+02 true resid norm 7.390530765626e+02 ||r(i)||/||b|| 2.140528789458e+00
>>       2 KSP preconditioned resid norm 7.111862680292e-04 true resid norm 4.066260265930e+02 ||r(i)||/||b|| 1.177716112777e+00
>>    42 SNES Function norm 3.452665901159e+02 
>>       0 KSP preconditioned resid norm 3.472654025442e+05 true resid norm 3.452665901159e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.552680791160e+02 true resid norm 1.064105325830e+03 ||r(i)||/||b|| 3.081981739016e+00
>>       2 KSP preconditioned resid norm 2.695346315896e-03 true resid norm 5.359619569435e+02 ||r(i)||/||b|| 1.552313407340e+00
>>    43 SNES Function norm 3.452665900083e+02 
>>       0 KSP preconditioned resid norm 2.823873507449e+05 true resid norm 3.452665900083e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.616335635707e+00 true resid norm 1.564312339233e+02 ||r(i)||/||b|| 4.530737651723e-01
>>       2 KSP preconditioned resid norm 1.580836053583e-04 true resid norm 1.777870114826e+02 ||r(i)||/||b|| 5.149267743465e-01
>>    44 SNES Function norm 3.452665895210e+02 
>>       0 KSP preconditioned resid norm 4.764215602672e+05 true resid norm 3.452665895210e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.491949190147e+01 true resid norm 4.652947485976e+02 ||r(i)||/||b|| 1.347639078670e+00
>>       2 KSP preconditioned resid norm 2.266106888017e-04 true resid norm 1.316722015429e+02 ||r(i)||/||b|| 3.813638664707e-01
>>    45 SNES Function norm 3.452665889075e+02 
>>       0 KSP preconditioned resid norm 1.439659890301e+05 true resid norm 3.452665889075e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.431461947162e+01 true resid norm 1.403614551576e+02 ||r(i)||/||b|| 4.065306625865e-01
>>       2 KSP preconditioned resid norm 4.711117728059e-04 true resid norm 1.095117712535e+02 ||r(i)||/||b|| 3.171803318704e-01
>>    46 SNES Function norm 3.452665876547e+02 
>>       0 KSP preconditioned resid norm 2.157313334766e+05 true resid norm 3.452665876547e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.967750174401e+00 true resid norm 1.053591658730e+02 ||r(i)||/||b|| 3.051530893525e-01
>>       2 KSP preconditioned resid norm 4.512059841816e-04 true resid norm 1.189061020519e+02 ||r(i)||/||b|| 3.443892525473e-01
>>    47 SNES Function norm 3.452665866229e+02 
>>       0 KSP preconditioned resid norm 3.981118078251e+05 true resid norm 3.452665866229e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.013119377232e+01 true resid norm 4.969789311062e+02 ||r(i)||/||b|| 1.439406390196e+00
>>       2 KSP preconditioned resid norm 1.872910444347e-04 true resid norm 1.396396719776e+02 ||r(i)||/||b|| 4.044401554852e-01
>>    48 SNES Function norm 3.452665850954e+02 
>>       0 KSP preconditioned resid norm 1.967554630062e+05 true resid norm 3.452665850954e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.006970327024e+00 true resid norm 9.616098906643e+01 ||r(i)||/||b|| 2.785122951874e-01
>>       2 KSP preconditioned resid norm 3.184368801558e-05 true resid norm 9.951979756268e+01 ||r(i)||/||b|| 2.882404549377e-01
>>    49 SNES Function norm 3.452665835600e+02 
>>       0 KSP preconditioned resid norm 3.965648247310e+05 true resid norm 3.452665835600e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.862969730098e+01 true resid norm 1.790616593203e+02 ||r(i)||/||b|| 5.186185627174e-01
>>       2 KSP preconditioned resid norm 2.964457441143e-04 true resid norm 1.447818294544e+02 ||r(i)||/||b|| 4.193334552146e-01
>>    50 SNES Function norm 3.452665828612e+02 
>>       0 KSP preconditioned resid norm 1.265022286683e+06 true resid norm 3.452665828612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.401066920177e+01 true resid norm 3.143671803790e+02 ||r(i)||/||b|| 9.105056671685e-01
>>       2 KSP preconditioned resid norm 1.444898193175e-04 true resid norm 7.578180059616e+01 ||r(i)||/||b|| 2.194877939480e-01
>>    51 SNES Function norm 3.452665817118e+02 
>>       0 KSP preconditioned resid norm 2.063334812849e+05 true resid norm 3.452665817118e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.074628697499e+01 true resid norm 1.155569679685e+02 ||r(i)||/||b|| 3.346891187543e-01
>>       2 KSP preconditioned resid norm 1.326420665890e-04 true resid norm 9.261090091148e+01 ||r(i)||/||b|| 2.682301323584e-01
>>    52 SNES Function norm 3.452665807811e+02 
>>       0 KSP preconditioned resid norm 6.549963238965e+06 true resid norm 3.452665807811e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.246352248726e+01 true resid norm 3.604213542889e+02 ||r(i)||/||b|| 1.043892963731e+00
>>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 52
>> 
>> which is a little different, but not so much different, than what i get if I use my Jacobian.
>> 
>> -gideon
>> 
>>> On Sep 8, 2015, at 10:00 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>> 
>>> Yup, now it?s working.
>>> -gideon
>>> 
>>>> On Sep 8, 2015, at 9:58 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>> 
>>>> On Tue, Sep 8, 2015 at 8:51 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>> Going back to this point, a case that previously worked, is now giving me problems that I don?t understand:
>>>> 
>>>>   0 SNES Function norm 1.421454390131e-02 
>>>>     0 KSP preconditioned resid norm 7.635339584553e-03 true resid norm 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 2.838956718317e-07 true resid norm 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
>>>>     2 KSP preconditioned resid norm 5.889748199467e-12 true resid norm 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
>>>>   1 SNES Function norm 4.476139939604e-06 
>>>>     0 KSP preconditioned resid norm 4.897102387844e-05 true resid norm 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.133649519960e-09 true resid norm 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
>>>>     2 KSP preconditioned resid norm 2.223328042462e-14 true resid norm 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
>>>>   2 SNES Function norm 5.447761863330e-09 
>>>>     0 KSP preconditioned resid norm 9.703890346186e-09 true resid norm 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 1.982949185218e-13 true resid norm 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
>>>>     2 KSP preconditioned resid norm 7.045835741315e-18 true resid norm 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
>>>> Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT iterations 2
>>>> 
>>>> Somehow the maximum number of function evaluations is being exceeded. Oh, are you using -snes_fd? This uses a function evaluation
>>>> for each column, so you might have to up it > 10000. The convergence is fine.
>>>> 
>>>>   Thanks,
>>>> 
>>>>      Matt
>>>>  
>>>> -gideon
>>>> 
>>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>> 
>>>>> 
>>>>>   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>>>> 
>>>>>  Barry
>>>>> 
>>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>> 
>>>>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>>>> 
>>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>>>>> 
>>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>>>> 
>>>>>>>   Matt
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>> 
>>>>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>>>> 
>>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>>>> 
>>>>>>>> Gideon,
>>>>>>>> 
>>>>>>>>  In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>>>> 
>>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -gideon
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>>>>> 
>>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>>>> 
>>>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>>>> 
>>>>>>>>>>>   0 SNES Function norm 3.857327250267e-09 
>>>>>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>>>   1 SNES Function norm 2.479697868412e-12 
>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>>>> 
>>>>>>>>>>> -gideon
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>>>> 
>>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>>>> 
>>>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>  0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>>>>  1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> then 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>>>>    type: seqaij
>>>>>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>>>      MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>>>>        PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>>>>          [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0]  143924 
>>>>>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>>>>          [0] 15991 
>>>>>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>>>    SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>>>    etc
>>>>>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/1e1a48bd/attachment-0001.html>

From amalsahai at gmail.com  Tue Sep  8 22:38:04 2015
From: amalsahai at gmail.com (Amal Sahai)
Date: Tue, 8 Sep 2015 22:38:04 -0500
Subject: [petsc-users] Setting absolute tolerance on ksp solver
Message-ID: <CA+8mhyV4VU29WdMUSr7qXkd6gA+1h9g7u3UG4O8Jtx68pe05Xw@mail.gmail.com>

I have been setting the tolerance for a ksp solver using KSPSetTolerances.
I would like to set my convergence criteria using only the absolute
convergence parameter. I use the following command to set this up:

KSPSetTolerances( ksp, PETSC_DEFAULT, atol, PETSC_DEFAULT, maxits)

The problem is my ksp solver considers only the value of rtol while
stopping the iterative solve. I tried changing the value of rtol to some
other parameter instead of PETSC_DEFAULT (while making sure that atol is
the more stringent condition requiring more iterations) but to no avail.

How can I fix this?

Regards
Amal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150908/96c0d50d/attachment.html>

From bsmith at mcs.anl.gov  Tue Sep  8 22:49:22 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 Sep 2015 22:49:22 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
Message-ID: <0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>


> On Sep 8, 2015, at 10:28 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I should clarify, in that run, I ran with the analytic jacobian, but I did not use -snes_mf_operator.  If I use that flag, the performance is a bit different.  In particular, the true residual norms are not as good.  How should I interpret that?

  The "analytic Jacobians" are wrong. Of course, that is easy to say, the hard part is figuring out exactly what entries are wrong.

  You can try the following options. They are experimental so may be flaky run with the options below (but not -snes_fd or -snes_mf_operators)

-snes_check_jacobian 
-snes_check_jacobian_view 

  Barry



> 
>       0 SNES Function norm 1.975693696426e-08 
>         0 KSP preconditioned resid norm 4.241029862559e-08 true resid norm 1.975693696426e-08 ||r(i)||/||b|| 1.000000000000e+00
>         1 KSP preconditioned resid norm 8.429547895868e-14 true resid norm 1.953386034503e-12 ||r(i)||/||b|| 9.887089471596e-05
>       1 SNES Function norm 2.220439366420e-12 
>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>     0 SNES Function norm 5.066222213175e+03 
>       0 KSP preconditioned resid norm 3.135087050116e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.763659023735e-04 true resid norm 2.125311397199e+00 ||r(i)||/||b|| 4.195061542448e-04
>       2 KSP preconditioned resid norm 1.084917879345e-09 true resid norm 9.170494798321e-03 ||r(i)||/||b|| 1.810124864731e-06
>     1 SNES Function norm 8.488370402237e+02 
>       0 KSP preconditioned resid norm 2.671196814418e+02 true resid norm 8.488370402237e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.493447352396e-03 true resid norm 1.977751087754e-01 ||r(i)||/||b|| 2.329953800358e-04
>       2 KSP preconditioned resid norm 2.764181028108e-08 true resid norm 1.635196652391e-01 ||r(i)||/||b|| 1.926396439957e-04
>     2 SNES Function norm 6.550793737252e+02 
>       0 KSP preconditioned resid norm 1.534706412676e+02 true resid norm 6.550793737252e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.638157030055e-03 true resid norm 8.038238857445e-02 ||r(i)||/||b|| 1.227063342223e-04
>       2 KSP preconditioned resid norm 4.712628487080e-08 true resid norm 7.077842320372e-02 ||r(i)||/||b|| 1.080455682816e-04
>     3 SNES Function norm 5.773378779059e+02 
>       0 KSP preconditioned resid norm 1.357060892902e+02 true resid norm 5.773378779059e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.956356650328e-02 true resid norm 6.547565039272e-01 ||r(i)||/||b|| 1.134095871731e-03
>       2 KSP preconditioned resid norm 4.061258280947e-08 true resid norm 6.285340893993e-02 ||r(i)||/||b|| 1.088676342663e-04
>     4 SNES Function norm 5.240279637579e+02 
>       0 KSP preconditioned resid norm 1.520925023264e+02 true resid norm 5.240279637579e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.439871073884e-02 true resid norm 7.370842666984e-01 ||r(i)||/||b|| 1.406574300754e-03
>       2 KSP preconditioned resid norm 2.929266069899e-08 true resid norm 6.966469677574e-02 ||r(i)||/||b|| 1.329408000981e-04
>     5 SNES Function norm 4.757123720235e+02 
>       0 KSP preconditioned resid norm 1.874598394237e+02 true resid norm 4.757123720235e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.666900422546e-02 true resid norm 4.366093312686e-01 ||r(i)||/||b|| 9.178010851630e-04
>       2 KSP preconditioned resid norm 1.569055410459e-07 true resid norm 8.415473672611e-02 ||r(i)||/||b|| 1.769025606127e-04
>     6 SNES Function norm 4.225299506849e+02 
>       0 KSP preconditioned resid norm 2.568609427837e+02 true resid norm 4.225299506849e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.408384520223e-02 true resid norm 5.711566149678e-01 ||r(i)||/||b|| 1.351754151491e-03
>       2 KSP preconditioned resid norm 1.091907927355e-07 true resid norm 1.205574809113e-01 ||r(i)||/||b|| 2.853229237735e-04
>     7 SNES Function norm 3.809903859846e+02 
>       0 KSP preconditioned resid norm 5.148606473696e+02 true resid norm 3.809903859846e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.519554844371e-02 true resid norm 5.230376280503e-01 ||r(i)||/||b|| 1.372836815025e-03
>       2 KSP preconditioned resid norm 7.972364126927e-07 true resid norm 2.637500099868e-01 ||r(i)||/||b|| 6.922747126680e-04
>     8 SNES Function norm 3.759602001862e+02 
>       0 KSP preconditioned resid norm 5.342247036495e+03 true resid norm 3.759602001862e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.133750669625e+00 true resid norm 1.703127877004e+01 ||r(i)||/||b|| 4.530074928571e-02
>       2 KSP preconditioned resid norm 4.996777563109e-06 true resid norm 2.867628968895e+00 ||r(i)||/||b|| 7.627480162728e-03
>     9 SNES Function norm 3.756183831703e+02 
>       0 KSP preconditioned resid norm 5.800146189851e+03 true resid norm 3.756183831703e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.928446833882e-01 true resid norm 6.567210094215e+00 ||r(i)||/||b|| 1.748372920086e-02
>       2 KSP preconditioned resid norm 3.716020308248e-06 true resid norm 3.211789561370e+00 ||r(i)||/||b|| 8.550671919361e-03
>    10 SNES Function norm 3.746165124359e+02 
>       0 KSP preconditioned resid norm 6.573797317895e+02 true resid norm 3.746165124359e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.168407014568e-01 true resid norm 1.224122498177e+00 ||r(i)||/||b|| 3.267668288879e-03
>       2 KSP preconditioned resid norm 6.626961415306e-07 true resid norm 3.840346541352e-01 ||r(i)||/||b|| 1.025140754309e-03
>    11 SNES Function norm 3.721805808051e+02 
>       0 KSP preconditioned resid norm 7.849993923903e+02 true resid norm 3.721805808051e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.857279790238e-02 true resid norm 5.226376723903e-01 ||r(i)||/||b|| 1.404258307244e-03
>       2 KSP preconditioned resid norm 6.283315938786e-07 true resid norm 4.622839625158e-01 ||r(i)||/||b|| 1.242095870547e-03
>    12 SNES Function norm 3.702012231848e+02 
>       0 KSP preconditioned resid norm 8.145587172693e+03 true resid norm 3.702012231848e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.293173409305e+00 true resid norm 1.444669645437e+01 ||r(i)||/||b|| 3.902390254166e-02
>       2 KSP preconditioned resid norm 5.333263851161e-06 true resid norm 4.631558046987e+00 ||r(i)||/||b|| 1.251092043171e-02
>    13 SNES Function norm 3.701675832926e+02 
>       0 KSP preconditioned resid norm 3.439618250319e+03 true resid norm 3.701675832926e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.424584259775e-01 true resid norm 9.488954827528e+00 ||r(i)||/||b|| 2.563421340984e-02
>       2 KSP preconditioned resid norm 2.361993681793e-06 true resid norm 2.151423050784e+00 ||r(i)||/||b|| 5.812024466453e-03
>    14 SNES Function norm 3.700043773583e+02 
>       0 KSP preconditioned resid norm 1.591977430573e+03 true resid norm 3.700043773583e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.627624143289e-01 true resid norm 1.937528552277e+00 ||r(i)||/||b|| 5.236501703332e-03
>       2 KSP preconditioned resid norm 1.406173142258e-06 true resid norm 9.301779435764e-01 ||r(i)||/||b|| 2.513964700141e-03
>    15 SNES Function norm 3.694535945999e+02 
>       0 KSP preconditioned resid norm 1.121321894103e+03 true resid norm 3.694535945999e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.897150400722e-01 true resid norm 2.095277723848e+00 ||r(i)||/||b|| 5.671287962747e-03
>       2 KSP preconditioned resid norm 7.510763366789e-07 true resid norm 6.477051057041e-01 ||r(i)||/||b|| 1.753143331588e-03
>    16 SNES Function norm 3.678972879038e+02 
>       0 KSP preconditioned resid norm 3.333082468735e+01 true resid norm 3.678972879038e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.431732430685e-03 true resid norm 1.616512305791e-02 ||r(i)||/||b|| 4.393922866356e-05
>       2 KSP preconditioned resid norm 6.434957827496e-08 true resid norm 1.918034542828e-02 ||r(i)||/||b|| 5.213505524211e-05
>    17 SNES Function norm 3.342697499638e+02 
>       0 KSP preconditioned resid norm 1.247233383993e+02 true resid norm 3.342697499638e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.479213016905e-02 true resid norm 2.646077519018e-01 ||r(i)||/||b|| 7.915994550225e-04
>       2 KSP preconditioned resid norm 1.491221634262e-07 true resid norm 7.551908725240e-02 ||r(i)||/||b|| 2.259225887493e-04
>    18 SNES Function norm 3.242724990245e+02 
>       0 KSP preconditioned resid norm 2.730129105212e+02 true resid norm 3.242724990245e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.298117117474e-02 true resid norm 2.331723933868e-01 ||r(i)||/||b|| 7.190631153991e-04
>       2 KSP preconditioned resid norm 2.245058389453e-07 true resid norm 1.607094606122e-01 ||r(i)||/||b|| 4.956000311333e-04
>    19 SNES Function norm 3.161198940824e+02 
>       0 KSP preconditioned resid norm 1.976453347207e+02 true resid norm 3.161198940824e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.069966316351e-02 true resid norm 3.606207026495e-01 ||r(i)||/||b|| 1.140771933055e-03
>       2 KSP preconditioned resid norm 1.339731244693e-06 true resid norm 1.150564760533e-01 ||r(i)||/||b|| 3.639646798796e-04
>    20 SNES Function norm 3.095294698084e+02 
>       0 KSP preconditioned resid norm 1.989229053011e+02 true resid norm 3.095294698084e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.509913758989e-02 true resid norm 3.251999852245e-01 ||r(i)||/||b|| 1.050626893219e-03
>       2 KSP preconditioned resid norm 3.119014445967e-07 true resid norm 1.169978920551e-01 ||r(i)||/||b|| 3.779862774539e-04
>    21 SNES Function norm 3.041156286651e+02 
>       0 KSP preconditioned resid norm 6.280117878327e+02 true resid norm 3.041156286651e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.168409567185e-01 true resid norm 1.048814657078e+00 ||r(i)||/||b|| 3.448736461464e-03
>       2 KSP preconditioned resid norm 1.609317462605e-06 true resid norm 3.730072311447e-01 ||r(i)||/||b|| 1.226530950684e-03
>    22 SNES Function norm 3.040366549628e+02 
>       0 KSP preconditioned resid norm 1.437979588458e+03 true resid norm 3.040366549628e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.585745252784e-01 true resid norm 3.748888651074e+00 ||r(i)||/||b|| 1.233038382011e-02
>       2 KSP preconditioned resid norm 2.280617561277e-06 true resid norm 8.938758394240e-01 ||r(i)||/||b|| 2.940026555460e-03
>    23 SNES Function norm 3.038811444081e+02 
>       0 KSP preconditioned resid norm 4.656588555628e+02 true resid norm 3.038811444081e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.379807174952e-01 true resid norm 1.502298425690e+00 ||r(i)||/||b|| 4.943703988664e-03
>       2 KSP preconditioned resid norm 1.371459645956e-06 true resid norm 2.736849872814e-01 ||r(i)||/||b|| 9.006316855046e-04
>    24 SNES Function norm 3.021353442803e+02 
>       0 KSP preconditioned resid norm 2.323079380432e+02 true resid norm 3.021353442803e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.154620938700e-01 true resid norm 1.429767871615e+00 ||r(i)||/||b|| 4.732209914140e-03
>       2 KSP preconditioned resid norm 7.092074512403e-07 true resid norm 1.370959604626e-01 ||r(i)||/||b|| 4.537567784038e-04
>    25 SNES Function norm 2.989453281461e+02 
>       0 KSP preconditioned resid norm 2.466046529201e+02 true resid norm 2.989453281461e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.542626606314e-01 true resid norm 1.672331292335e+00 ||r(i)||/||b|| 5.594104121668e-03
>       2 KSP preconditioned resid norm 8.410454494090e-07 true resid norm 1.432702712514e-01 ||r(i)||/||b|| 4.792524176238e-04
>    26 SNES Function norm 2.965575466120e+02 
>       0 KSP preconditioned resid norm 1.729310576069e+02 true resid norm 2.965575466120e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.667588027964e-02 true resid norm 9.915795006833e-01 ||r(i)||/||b|| 3.343632667628e-03
>       2 KSP preconditioned resid norm 5.726421091521e-07 true resid norm 1.042312787918e-01 ||r(i)||/||b|| 3.514706672706e-04
>    27 SNES Function norm 2.886519975216e+02 
>       0 KSP preconditioned resid norm 3.474883023572e+02 true resid norm 2.886519975216e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.449042223312e-01 true resid norm 2.910159944511e+00 ||r(i)||/||b|| 1.008189781986e-02
>       2 KSP preconditioned resid norm 4.054998693233e-06 true resid norm 2.177883931275e-01 ||r(i)||/||b|| 7.545015970699e-04
>    28 SNES Function norm 2.877765511463e+02 
>       0 KSP preconditioned resid norm 6.932304384488e+02 true resid norm 2.877765511463e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.623309918707e-01 true resid norm 6.631612349599e+00 ||r(i)||/||b|| 2.304431102251e-02
>       2 KSP preconditioned resid norm 8.935815040540e-06 true resid norm 4.208867273501e-01 ||r(i)||/||b|| 1.462546985408e-03
>    29 SNES Function norm 2.874829836383e+02 
>       0 KSP preconditioned resid norm 3.031373648629e+02 true resid norm 2.874829836383e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.097809781088e-01 true resid norm 8.269361024084e-01 ||r(i)||/||b|| 2.876469737244e-03
>       2 KSP preconditioned resid norm 1.242142566726e-06 true resid norm 1.780035410476e-01 ||r(i)||/||b|| 6.191793990544e-04
>    30 SNES Function norm 2.857592509565e+02 
>       0 KSP preconditioned resid norm 2.905142832215e+02 true resid norm 2.857592509565e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.993930292806e-02 true resid norm 4.363268013941e-01 ||r(i)||/||b|| 1.526903503329e-03
>       2 KSP preconditioned resid norm 1.565987527263e-06 true resid norm 1.752909971699e-01 ||r(i)||/||b|| 6.134219507616e-04
>    31 SNES Function norm 2.835404531301e+02 
>       0 KSP preconditioned resid norm 1.787845486677e+02 true resid norm 2.835404531301e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.133510370229e-01 true resid norm 1.954968834465e+00 ||r(i)||/||b|| 6.894849792627e-03
>       2 KSP preconditioned resid norm 1.325071802836e-06 true resid norm 1.055727452178e-01 ||r(i)||/||b|| 3.723375061738e-04
>    32 SNES Function norm 2.766251793953e+02 
>       0 KSP preconditioned resid norm 8.373384127615e+01 true resid norm 2.766251793953e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.599342282394e-02 true resid norm 1.296081686745e-01 ||r(i)||/||b|| 4.685335187412e-04
>       2 KSP preconditioned resid norm 8.378945187131e-08 true resid norm 4.905876720151e-02 ||r(i)||/||b|| 1.773474392633e-04
>    33 SNES Function norm 2.763417748035e+02 
>       0 KSP preconditioned resid norm 4.315380927138e+02 true resid norm 2.763417748035e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.240956674451e-02 true resid norm 8.440426813509e-01 ||r(i)||/||b|| 3.054343419307e-03
>       2 KSP preconditioned resid norm 1.055259274458e-06 true resid norm 2.556138363110e-01 ||r(i)||/||b|| 9.249916574963e-04
>    34 SNES Function norm 2.636632257368e+02 
>       0 KSP preconditioned resid norm 2.002447589825e+02 true resid norm 2.636632257368e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.091871038526e-01 true resid norm 6.938948038720e-01 ||r(i)||/||b|| 2.631746622734e-03
>       2 KSP preconditioned resid norm 1.078660902246e-06 true resid norm 1.161053485203e-01 ||r(i)||/||b|| 4.403547297725e-04
>    35 SNES Function norm 2.607600958607e+02 
>       0 KSP preconditioned resid norm 1.036034642027e+02 true resid norm 2.607600958607e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.431459913062e-02 true resid norm 2.275229425112e-01 ||r(i)||/||b|| 8.725374247171e-04
>       2 KSP preconditioned resid norm 2.358846414843e-07 true resid norm 6.052444521984e-02 ||r(i)||/||b|| 2.321077733158e-04
>    36 SNES Function norm 2.559001581389e+02 
>       0 KSP preconditioned resid norm 1.549260221638e+02 true resid norm 2.559001581389e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.556726070406e-01 true resid norm 1.861290598804e+00 ||r(i)||/||b|| 7.273503120674e-03
>       2 KSP preconditioned resid norm 1.968725491683e-06 true resid norm 9.363655703756e-02 ||r(i)||/||b|| 3.659105086865e-04
>    37 SNES Function norm 2.518397862326e+02 
>       0 KSP preconditioned resid norm 5.597954308908e+01 true resid norm 2.518397862326e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.026654037349e-02 true resid norm 1.814101931154e-01 ||r(i)||/||b|| 7.203396882964e-04
>       2 KSP preconditioned resid norm 6.253768805342e-08 true resid norm 3.229917855960e-02 ||r(i)||/||b|| 1.282528826870e-04
>    38 SNES Function norm 2.316134843390e+02 
>       0 KSP preconditioned resid norm 7.012435360694e+02 true resid norm 2.316134843390e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.076320346419e-01 true resid norm 4.417669735438e+00 ||r(i)||/||b|| 1.907345657377e-02
>       2 KSP preconditioned resid norm 1.250819622244e-06 true resid norm 4.042382807831e-01 ||r(i)||/||b|| 1.745314103524e-03
>    39 SNES Function norm 2.315003516125e+02 
>       0 KSP preconditioned resid norm 1.770628877633e+03 true resid norm 2.315003516125e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.699599407714e-01 true resid norm 5.719451377836e+00 ||r(i)||/||b|| 2.470601594337e-02
>       2 KSP preconditioned resid norm 1.342291667594e-05 true resid norm 1.009488471262e+00 ||r(i)||/||b|| 4.360634721416e-03
>    40 SNES Function norm 2.314745966456e+02 
>       0 KSP preconditioned resid norm 3.814990651391e+03 true resid norm 2.314745966456e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.487459394941e+00 true resid norm 3.985946817773e+01 ||r(i)||/||b|| 1.721980241260e-01
>       2 KSP preconditioned resid norm 1.472896798351e-05 true resid norm 2.616262470753e+00 ||r(i)||/||b|| 1.130259004083e-02
>    41 SNES Function norm 2.314707133124e+02 
>       0 KSP preconditioned resid norm 3.619965563807e+04 true resid norm 2.314707133124e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.046216042689e+01 true resid norm 1.485715627898e+02 ||r(i)||/||b|| 6.418590095641e-01
>       2 KSP preconditioned resid norm 5.088996886285e-05 true resid norm 7.571617793696e+00 ||r(i)||/||b|| 3.271091053095e-02
>    42 SNES Function norm 2.314704913770e+02 
>       0 KSP preconditioned resid norm 6.684430447231e+03 true resid norm 2.314704913770e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.035066002617e+00 true resid norm 1.498611774406e+01 ||r(i)||/||b|| 6.474310247889e-02
>       2 KSP preconditioned resid norm 1.894517227201e-05 true resid norm 3.620274714957e+00 ||r(i)||/||b|| 1.564032932846e-02
>    43 SNES Function norm 2.314685063519e+02 
>       0 KSP preconditioned resid norm 1.519862455568e+04 true resid norm 2.314685063519e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.799545478657e+02 true resid norm 1.315130172649e+03 ||r(i)||/||b|| 5.681680818597e+00
>       2 KSP preconditioned resid norm 7.538897078155e-04 true resid norm 5.814357004150e+01 ||r(i)||/||b|| 2.511943026629e-01
>    44 SNES Function norm 2.314684987281e+02 
>       0 KSP preconditioned resid norm 1.914858613894e+04 true resid norm 2.314684987281e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.610361797677e+00 true resid norm 2.904545821866e+01 ||r(i)||/||b|| 1.254834173041e-01
>       2 KSP preconditioned resid norm 8.459090941337e-06 true resid norm 9.804924579099e+00 ||r(i)||/||b|| 4.235964994363e-02
>    45 SNES Function norm 2.314682724575e+02 
>       0 KSP preconditioned resid norm 1.628064355012e+04 true resid norm 2.314682724575e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.192253277458e+01 true resid norm 8.717348030044e+01 ||r(i)||/||b|| 3.766109254409e-01
>       2 KSP preconditioned resid norm 2.901075005128e-05 true resid norm 5.898778957654e+00 ||r(i)||/||b|| 2.548417930037e-02
>    46 SNES Function norm 2.314675179712e+02 
>       0 KSP preconditioned resid norm 1.393728659990e+04 true resid norm 2.314675179712e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.363450892258e+01 true resid norm 9.876887689991e+01 ||r(i)||/||b|| 4.267072882002e-01
>       2 KSP preconditioned resid norm 3.348633706956e-05 true resid norm 4.651974796776e+00 ||r(i)||/||b|| 2.009774346548e-02
>    47 SNES Function norm 2.314663411831e+02 
>       0 KSP preconditioned resid norm 7.856043279131e+03 true resid norm 2.314663411831e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.677460009854e+00 true resid norm 2.021525646408e+01 ||r(i)||/||b|| 8.733562020616e-02
>       2 KSP preconditioned resid norm 4.033484713601e-05 true resid norm 4.191927270226e+00 ||r(i)||/||b|| 1.811031033194e-02
>    48 SNES Function norm 2.314648869203e+02 
>       0 KSP preconditioned resid norm 1.052613791778e+04 true resid norm 2.314648869203e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.177731181359e+00 true resid norm 2.344482867012e+01 ||r(i)||/||b|| 1.012889211061e-01
>       2 KSP preconditioned resid norm 4.337358847947e-05 true resid norm 5.365943818938e+00 ||r(i)||/||b|| 2.318253921937e-02
>    49 SNES Function norm 2.314640197245e+02 
>       0 KSP preconditioned resid norm 2.047116917526e+04 true resid norm 2.314640197245e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.560093775943e+00 true resid norm 7.015143565198e+01 ||r(i)||/||b|| 3.030770645713e-01
>       2 KSP preconditioned resid norm 5.024217804080e-05 true resid norm 8.482232519276e+00 ||r(i)||/||b|| 3.664600886726e-02
>    50 SNES Function norm 2.314636382129e+02 
>       0 KSP preconditioned resid norm 2.202428764601e+04 true resid norm 2.314636382129e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.644800264238e+02 true resid norm 2.634313227181e+03 ||r(i)||/||b|| 1.138111043065e+01
>       2 KSP preconditioned resid norm 2.914035192146e-03 true resid norm 1.866099396017e+02 ||r(i)||/||b|| 8.062170846464e-01
>    51 SNES Function norm 2.314636367493e+02 
>       0 KSP preconditioned resid norm 1.948632462873e+04 true resid norm 2.314636367493e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.300034176850e+01 true resid norm 9.512460873903e+01 ||r(i)||/||b|| 4.109699911182e-01
>       2 KSP preconditioned resid norm 5.189046693053e-05 true resid norm 6.633939889168e+00 ||r(i)||/||b|| 2.866082976288e-02
>    52 SNES Function norm 2.314630481752e+02 
>       0 KSP preconditioned resid norm 1.491442414085e+04 true resid norm 2.314630481752e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.379241745164e+01 true resid norm 1.007276964292e+02 ||r(i)||/||b|| 4.351783026420e-01
>       2 KSP preconditioned resid norm 4.161679438843e-05 true resid norm 4.977179034848e+00 ||r(i)||/||b|| 2.150312576494e-02
>    53 SNES Function norm 2.314619494854e+02 
>       0 KSP preconditioned resid norm 1.072804170247e+04 true resid norm 2.314619494854e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.968714022053e+00 true resid norm 1.493125196687e+01 ||r(i)||/||b|| 6.450845160541e-02
>       2 KSP preconditioned resid norm 9.504384832756e-06 true resid norm 5.788047965007e+00 ||r(i)||/||b|| 2.500647721094e-02
>    54 SNES Function norm 2.314611811294e+02 
>       0 KSP preconditioned resid norm 2.352264129011e+04 true resid norm 2.314611811294e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.347783610050e+01 true resid norm 9.827481962397e+01 ||r(i)||/||b|| 4.245844557797e-01
>       2 KSP preconditioned resid norm 2.918515899364e-05 true resid norm 7.891003112752e+00 ||r(i)||/||b|| 3.409212324177e-02
>    55 SNES Function norm 2.314607568296e+02 
>       0 KSP preconditioned resid norm 1.749283175470e+04 true resid norm 2.314607568296e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.070162653778e+00 true resid norm 2.389094964330e+01 ||r(i)||/||b|| 1.032181436307e-01
>       2 KSP preconditioned resid norm 1.631144493664e-05 true resid norm 1.139801540267e+01 ||r(i)||/||b|| 4.924383536454e-02
>    56 SNES Function norm 2.314605438312e+02 
>       0 KSP preconditioned resid norm 7.109914723607e+04 true resid norm 2.314605438312e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.363804435455e+01 true resid norm 6.067805272857e+02 ||r(i)||/||b|| 2.621528996874e+00
>       2 KSP preconditioned resid norm 3.256368262965e-04 true resid norm 9.390634078085e+01 ||r(i)||/||b|| 4.057120890952e-01
>    57 SNES Function norm 2.314605414352e+02 
>       0 KSP preconditioned resid norm 2.888647244791e+04 true resid norm 2.314605414352e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.096880192515e+01 true resid norm 8.143424079389e+01 ||r(i)||/||b|| 3.518277469194e-01
>       2 KSP preconditioned resid norm 8.022062493403e-05 true resid norm 1.077904577963e+01 ||r(i)||/||b|| 4.656969050877e-02
>    58 SNES Function norm 2.314603336141e+02 
>       0 KSP preconditioned resid norm 1.836068875705e+04 true resid norm 2.314603336141e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 9.090981797341e+01 true resid norm 6.587841779043e+02 ||r(i)||/||b|| 2.846207674628e+00
>       2 KSP preconditioned resid norm 8.275701701453e-04 true resid norm 4.183336915180e+01 ||r(i)||/||b|| 1.807366666184e-01
>    59 SNES Function norm 2.314603179098e+02 
>       0 KSP preconditioned resid norm 2.853255155449e+04 true resid norm 2.314603179098e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.327796718536e+01 true resid norm 4.583228284114e+02 ||r(i)||/||b|| 1.980135655866e+00
>       2 KSP preconditioned resid norm 5.951560456364e-04 true resid norm 5.003183584303e+01 ||r(i)||/||b|| 2.161572933747e-01
>    60 SNES Function norm 2.314603080054e+02 
>       0 KSP preconditioned resid norm 7.163155165743e+04 true resid norm 2.314603080054e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.633016255887e+01 true resid norm 3.379504724720e+02 ||r(i)||/||b|| 1.460079593708e+00
>       2 KSP preconditioned resid norm 1.403149036303e-04 true resid norm 2.609855476909e+01 ||r(i)||/||b|| 1.127560703344e-01
>    61 SNES Function norm 2.314602358658e+02 
>       0 KSP preconditioned resid norm 1.802787930999e+04 true resid norm 2.314602358658e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.381933291542e+01 true resid norm 1.003023231896e+02 ||r(i)||/||b|| 4.333458091167e-01
>       2 KSP preconditioned resid norm 1.089082641399e-04 true resid norm 5.919741402584e+00 ||r(i)||/||b|| 2.557563021761e-02
>    62 SNES Function norm 2.314594914718e+02 
>       0 KSP preconditioned resid norm 1.293341944019e+04 true resid norm 2.314594914718e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.507379326494e+00 true resid norm 1.900387953811e+01 ||r(i)||/||b|| 8.210455927845e-02
>       2 KSP preconditioned resid norm 1.049822021718e-05 true resid norm 8.049347657678e+00 ||r(i)||/||b|| 3.477648553746e-02
>    63 SNES Function norm 2.314590886372e+02 
>       0 KSP preconditioned resid norm 5.233660621212e+04 true resid norm 2.314590886372e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.767828625949e+01 true resid norm 4.972067876731e+02 ||r(i)||/||b|| 2.148141127664e+00
>       2 KSP preconditioned resid norm 1.565714280046e-03 true resid norm 9.541512803665e+01 ||r(i)||/||b|| 4.122332313604e-01
>    64 SNES Function norm 2.314590857486e+02 
>       0 KSP preconditioned resid norm 9.223438894260e+04 true resid norm 2.314590857486e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.771340682426e+01 true resid norm 1.439668879180e+02 ||r(i)||/||b|| 6.219971337587e-01
>       2 KSP preconditioned resid norm 4.313497493249e-04 true resid norm 8.319281395963e+01 ||r(i)||/||b|| 3.594277307826e-01
>    65 SNES Function norm 2.314590820052e+02 
>       0 KSP preconditioned resid norm 2.716078594523e+06 true resid norm 2.314590820052e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.132140332876e+01 true resid norm 2.279043369448e+02 ||r(i)||/||b|| 9.846420152120e-01
>       2 KSP preconditioned resid norm 3.421563776706e-04 true resid norm 3.596175647535e+01 ||r(i)||/||b|| 1.553698224489e-01
>    66 SNES Function norm 2.314590616296e+02 
>       0 KSP preconditioned resid norm 4.202813913686e+04 true resid norm 2.314590616296e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.895156465866e+00 true resid norm 5.779056270257e+01 ||r(i)||/||b|| 2.496794132651e-01
>       2 KSP preconditioned resid norm 1.534565623678e-04 true resid norm 1.837098704611e+01 ||r(i)||/||b|| 7.937035135615e-02
>    67 SNES Function norm 2.314589864333e+02 
>       0 KSP preconditioned resid norm 4.021695169087e+04 true resid norm 2.314589864333e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.852569690755e+01 true resid norm 3.590909196842e+02 ||r(i)||/||b|| 1.551423538216e+00
>       2 KSP preconditioned resid norm 1.348973307542e-04 true resid norm 6.035401149038e+01 ||r(i)||/||b|| 2.607546694143e-01
>    68 SNES Function norm 2.314589801992e+02 
>       0 KSP preconditioned resid norm 1.725952032453e+05 true resid norm 2.314589801992e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.777975072103e+01 true resid norm 3.518023660349e+02 ||r(i)||/||b|| 1.519933967272e+00
>       2 KSP preconditioned resid norm 1.266007649822e-04 true resid norm 7.313840992175e+01 ||r(i)||/||b|| 3.159886467089e-01
>    69 SNES Function norm 2.314589719235e+02 
>       0 KSP preconditioned resid norm 3.607076901939e+04 true resid norm 2.314589719235e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.210132261593e+02 true resid norm 8.774444290589e+02 ||r(i)||/||b|| 3.790928568321e+00
>       2 KSP preconditioned resid norm 4.891015819530e-04 true resid norm 8.098219226779e+01 ||r(i)||/||b|| 3.498770930970e-01
>    70 SNES Function norm 2.314589643803e+02 
>       0 KSP preconditioned resid norm 2.442776989201e+04 true resid norm 2.314589643803e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.014921648500e+02 true resid norm 2.188843891071e+03 ||r(i)||/||b|| 9.456725501782e+00
>       2 KSP preconditioned resid norm 1.771597678165e-03 true resid norm 1.481761731272e+02 ||r(i)||/||b|| 6.401833410254e-01
>    71 SNES Function norm 2.314589631504e+02 
>       0 KSP preconditioned resid norm 2.803938255445e+04 true resid norm 2.314589631504e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.937985936300e+01 true resid norm 5.033022630111e+02 ||r(i)||/||b|| 2.174477307600e+00
>       2 KSP preconditioned resid norm 3.691607880973e-04 true resid norm 2.715686837331e+01 ||r(i)||/||b|| 1.173290850511e-01
>    72 SNES Function norm 2.314589440605e+02 
>       0 KSP preconditioned resid norm 9.069257939432e+03 true resid norm 2.314589440605e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.034392926329e+01 true resid norm 7.627620750651e+01 ||r(i)||/||b|| 3.295453015052e-01
>       2 KSP preconditioned resid norm 2.062351156668e-04 true resid norm 6.951980961959e+00 ||r(i)||/||b|| 3.003548206001e-02
>    73 SNES Function norm 2.314585545983e+02 
>       0 KSP preconditioned resid norm 1.678113228786e+04 true resid norm 2.314585545983e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.926568185998e+01 true resid norm 2.132342065400e+02 ||r(i)||/||b|| 9.212630179515e-01
>       2 KSP preconditioned resid norm 7.165391508181e-05 true resid norm 1.872154768117e+01 ||r(i)||/||b|| 8.088509717718e-02
>    74 SNES Function norm 2.314584904582e+02 
>       0 KSP preconditioned resid norm 1.866490154072e+06 true resid norm 2.314584904582e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.383059477203e+01 true resid norm 2.450077515274e+02 ||r(i)||/||b|| 1.058538621946e+00
>       2 KSP preconditioned resid norm 1.572755583893e-04 true resid norm 6.900003764147e+01 ||r(i)||/||b|| 2.981097712375e-01
>    75 SNES Function norm 2.314584801547e+02 
>       0 KSP preconditioned resid norm 6.156188967138e+04 true resid norm 2.314584801547e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.447913103987e+01 true resid norm 1.072790720471e+02 ||r(i)||/||b|| 4.634916464304e-01
>       2 KSP preconditioned resid norm 5.644002952175e-05 true resid norm 1.929574883131e+01 ||r(i)||/||b|| 8.336591866678e-02
>    76 SNES Function norm 2.314584104532e+02 
>       0 KSP preconditioned resid norm 4.115894770006e+04 true resid norm 2.314584104532e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.929751893305e+01 true resid norm 3.567001584633e+02 ||r(i)||/||b|| 1.541098280960e+00
>       2 KSP preconditioned resid norm 4.520979883364e-04 true resid norm 6.072093527410e+01 ||r(i)||/||b|| 2.623405870420e-01
>    77 SNES Function norm 2.314584051144e+02 
>       0 KSP preconditioned resid norm 2.095491764620e+05 true resid norm 2.314584051144e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.552674762551e+01 true resid norm 2.582375444470e+02 ||r(i)||/||b|| 1.115697415781e+00
>       2 KSP preconditioned resid norm 7.435164108048e-05 true resid norm 1.988766346637e+01 ||r(i)||/||b|| 8.592327185759e-02
>    78 SNES Function norm 2.314582888096e+02 
>       0 KSP preconditioned resid norm 1.294622386441e+04 true resid norm 2.314582888096e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.107411018281e+01 true resid norm 8.058835795266e+01 ||r(i)||/||b|| 3.481765909838e-01
>       2 KSP preconditioned resid norm 3.309588564016e-05 true resid norm 4.903075273077e+00 ||r(i)||/||b|| 2.118340759493e-02
>    79 SNES Function norm 2.314571909137e+02 
>       0 KSP preconditioned resid norm 1.204264280969e+04 true resid norm 2.314571909137e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.737477639497e+00 true resid norm 1.301274832418e+01 ||r(i)||/||b|| 5.622097232241e-02
>       2 KSP preconditioned resid norm 1.116891328813e-05 true resid norm 6.616099462928e+00 ||r(i)||/||b|| 2.858454920674e-02
>    80 SNES Function norm 2.314565930335e+02 
>       0 KSP preconditioned resid norm 2.685765309163e+04 true resid norm 2.314565930335e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.392905224116e+01 true resid norm 1.017442509084e+02 ||r(i)||/||b|| 4.395824269896e-01
>       2 KSP preconditioned resid norm 1.415949639751e-04 true resid norm 2.252094414009e+01 ||r(i)||/||b|| 9.730094029692e-02
>    81 SNES Function norm 2.314565401448e+02 
>       0 KSP preconditioned resid norm 4.814419294191e+05 true resid norm 2.314565401448e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 5.857262358676e+01 true resid norm 5.054713713870e+02 ||r(i)||/||b|| 2.183871629079e+00
>       2 KSP preconditioned resid norm 1.243507216932e-03 true resid norm 3.772098143596e+02 ||r(i)||/||b|| 1.629721994996e+00
>    82 SNES Function norm 2.314565398085e+02 
>       0 KSP preconditioned resid norm 2.026301569876e+05 true resid norm 2.314565398085e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.987037388147e+01 true resid norm 2.882471210283e+02 ||r(i)||/||b|| 1.245361748114e+00
>       2 KSP preconditioned resid norm 2.169665365314e-04 true resid norm 4.035341344938e+01 ||r(i)||/||b|| 1.743455314884e-01
>    83 SNES Function norm 2.314565078487e+02 
>       0 KSP preconditioned resid norm 3.094380381349e+04 true resid norm 2.314565078487e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.978293556536e+00 true resid norm 2.635581568290e+01 ||r(i)||/||b|| 1.138694086758e-01
>       2 KSP preconditioned resid norm 1.890022323482e-05 true resid norm 1.597791541658e+01 ||r(i)||/||b|| 6.903204219699e-02
>    84 SNES Function norm 2.314564079163e+02 
>       0 KSP preconditioned resid norm 6.209269840680e+04 true resid norm 2.314564079163e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.995982106142e+01 true resid norm 1.468548855791e+02 ||r(i)||/||b|| 6.344818313788e-01
>       2 KSP preconditioned resid norm 7.230034458440e-05 true resid norm 1.277927519154e+01 ||r(i)||/||b|| 5.521244931858e-02
>    85 SNES Function norm 2.314562581879e+02 
>       0 KSP preconditioned resid norm 2.316871639948e+04 true resid norm 2.314562581879e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.061899183336e+01 true resid norm 7.658836412307e+01 ||r(i)||/||b|| 3.308977891662e-01
>       2 KSP preconditioned resid norm 1.593652837122e-04 true resid norm 8.955355396819e+00 ||r(i)||/||b|| 3.869135130297e-02
>    86 SNES Function norm 2.314559345442e+02 
>       0 KSP preconditioned resid norm 2.271158654394e+04 true resid norm 2.314559345442e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.441411557989e+01 true resid norm 1.074834857947e+02 ||r(i)||/||b|| 4.643799088859e-01
>       2 KSP preconditioned resid norm 1.535735640673e-04 true resid norm 1.965949694313e+01 ||r(i)||/||b|| 8.493840083143e-02
>    87 SNES Function norm 2.314559150625e+02 
>       0 KSP preconditioned resid norm 3.119665022860e+04 true resid norm 2.314559150625e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.487979701070e+01 true resid norm 1.076952042274e+02 ||r(i)||/||b|| 4.652946726303e-01
>       2 KSP preconditioned resid norm 2.534108740613e-04 true resid norm 2.577373574045e+01 ||r(i)||/||b|| 1.113548371986e-01
>    88 SNES Function norm 2.314558771417e+02 
>       0 KSP preconditioned resid norm 4.567599584574e+05 true resid norm 2.314558771417e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.380099279901e+00 true resid norm 1.760087395396e+02 ||r(i)||/||b|| 7.604418678547e-01
>       2 KSP preconditioned resid norm 1.693278102818e-03 true resid norm 2.175019651802e+02 ||r(i)||/||b|| 9.397124318735e-01
>    89 SNES Function norm 2.314558765490e+02 
>       0 KSP preconditioned resid norm 6.893967531875e+05 true resid norm 2.314558765490e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.982785897195e+01 true resid norm 2.190225259947e+02 ||r(i)||/||b|| 9.462819836779e-01
>       2 KSP preconditioned resid norm 2.006267159571e-04 true resid norm 2.365026583574e+01 ||r(i)||/||b|| 1.021804509281e-01
>    90 SNES Function norm 2.314558316055e+02 
>       0 KSP preconditioned resid norm 3.213811396790e+04 true resid norm 2.314558316055e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.863726299685e+01 true resid norm 1.346365978970e+02 ||r(i)||/||b|| 5.816945590139e-01
>       2 KSP preconditioned resid norm 8.119950416130e-05 true resid norm 7.914584107016e+00 ||r(i)||/||b|| 3.419479238054e-02
>    91 SNES Function norm 2.314553994731e+02 
>       0 KSP preconditioned resid norm 1.428885884403e+04 true resid norm 2.314553994731e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 6.461923684043e+00 true resid norm 4.652208559908e+01 ||r(i)||/||b|| 2.009980570986e-01
>       2 KSP preconditioned resid norm 6.018427586116e-05 true resid norm 6.443132757602e+00 ||r(i)||/||b|| 2.783747007963e-02
>    92 SNES Function norm 2.314548022234e+02 
>       0 KSP preconditioned resid norm 1.997882268842e+04 true resid norm 2.314548022234e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.990540367836e+01 true resid norm 1.444167499553e+02 ||r(i)||/||b|| 6.239522730484e-01
>       2 KSP preconditioned resid norm 6.933227199420e-05 true resid norm 4.340884186507e+00 ||r(i)||/||b|| 1.875478125667e-02
>    93 SNES Function norm 2.314535554897e+02 
>       0 KSP preconditioned resid norm 5.317751765458e+03 true resid norm 2.314535554897e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.051772818836e+01 true resid norm 7.626983587464e+01 ||r(i)||/||b|| 3.295254450219e-01
>       2 KSP preconditioned resid norm 1.062889530517e-04 true resid norm 4.148559709642e+00 ||r(i)||/||b|| 1.792394029491e-02
>    94 SNES Function norm 2.314519884611e+02 
>       0 KSP preconditioned resid norm 8.518590251100e+04 true resid norm 2.314519884611e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.400567581249e+01 true resid norm 5.350691354513e+02 ||r(i)||/||b|| 2.311793210371e+00
>       2 KSP preconditioned resid norm 3.190984674306e-04 true resid norm 9.080604176445e+01 ||r(i)||/||b|| 3.923320873940e-01
>    95 SNES Function norm 2.314519821638e+02 
>       0 KSP preconditioned resid norm 3.971419728627e+04 true resid norm 2.314519821638e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.584583085310e+02 true resid norm 1.155052020747e+03 ||r(i)||/||b|| 4.990460699229e+00
>       2 KSP preconditioned resid norm 8.131629823886e-04 true resid norm 1.353955847914e+02 ||r(i)||/||b|| 5.849834748684e-01
>    96 SNES Function norm 2.314519808162e+02 
>       0 KSP preconditioned resid norm 5.113274886640e+04 true resid norm 2.314519808162e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 8.658920186396e+01 true resid norm 6.338848251652e+02 ||r(i)||/||b|| 2.738731476524e+00
>       2 KSP preconditioned resid norm 6.225448401968e-04 true resid norm 7.231481532235e+01 ||r(i)||/||b|| 3.124398204212e-01
>    97 SNES Function norm 2.314519732830e+02 
>       0 KSP preconditioned resid norm 2.374489458388e+04 true resid norm 2.314519732830e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 7.056237350775e+00 true resid norm 5.327919633484e+01 ||r(i)||/||b|| 2.301954724305e-01
>       2 KSP preconditioned resid norm 8.341487773131e-05 true resid norm 1.097883645826e+01 ||r(i)||/||b|| 4.743462024773e-02
>    98 SNES Function norm 2.314517417065e+02 
>       0 KSP preconditioned resid norm 2.725936939754e+04 true resid norm 2.314517417065e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 4.470892119649e+00 true resid norm 3.482468106411e+01 ||r(i)||/||b|| 1.504619529209e-01
>       2 KSP preconditioned resid norm 2.743947437756e-05 true resid norm 1.930558512719e+01 ||r(i)||/||b|| 8.341084402670e-02
>    99 SNES Function norm 2.314516598045e+02 
>       0 KSP preconditioned resid norm 1.323063699793e+05 true resid norm 2.314516598045e+02 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 2.183745290203e+03 true resid norm 1.606786653196e+04 ||r(i)||/||b|| 6.942212704601e+01
>       2 KSP preconditioned resid norm 1.339289871939e-02 true resid norm 7.097265620768e+03 ||r(i)||/||b|| 3.066413793170e+01
>   100 SNES Function norm 2.314516598037e+02 
>   Nonlinear solve did not converge due to DIVERGED_MAX_IT iterations 100
> -gideon
> 
>> On Sep 8, 2015, at 10:53 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Tue, Sep 8, 2015 at 9:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> Your comment about the different pieces scaling different made me think of something, and I made a slight modification.  It still fails, but the conditioning looks much better, no?
>> 
>> This is great. Now its clear that the nonlinear convergence is crappy.
>> 
>> Lets start with looking at the distribution of residuals. Take iterate 50,
>> which is as converged as you get and save the residual (VecView with
>> binary). Then we can load it up and play with it. I recommend
>> 
>>   a) Norms for both halves
>> 
>>   b) Plot of the big half
>> 
>> What usually happens is that some small part is screwing up, but you
>> have to iterate the whole system, and finding the descent direction is
>> hard. If you select the problem variables, Newton will converge, and then
>> the big system will converge once the problem is removed. This is the classic
>> "coarse basis selection" problem in the nonlinear regime.
>> 
>>    Matt
>> 
>>   Thanks,
>> 
>>     Matt
>>  
>> 
>>       0 SNES Function norm 4.027474333769e-09 
>>         0 KSP preconditioned resid norm 9.390682304142e-09 true resid norm 4.027474333769e-09 ||r(i)||/||b|| 1.000000000000e+00
>>         1 KSP preconditioned resid norm 4.937459249116e-20 true resid norm 3.737772204700e-19 ||r(i)||/||b|| 9.280685350023e-11
>>       1 SNES Function norm 2.666655509899e-12 
>>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>     0 SNES Function norm 5.066222213175e+03 
>>       0 KSP preconditioned resid norm 3.135087050015e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.220576053756e-11 true resid norm 1.206309587478e-07 ||r(i)||/||b|| 2.381083056997e-11
>>     1 SNES Function norm 8.484697184819e+02 
>>       0 KSP preconditioned resid norm 2.668083368240e+02 true resid norm 8.484697184819e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.388166746665e-10 true resid norm 2.735732864000e-08 ||r(i)||/||b|| 3.224314085004e-11
>>     2 SNES Function norm 6.549559723821e+02 
>>       0 KSP preconditioned resid norm 1.552154273063e+02 true resid norm 6.549559723821e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.740227244709e-10 true resid norm 1.800722435354e-08 ||r(i)||/||b|| 2.749379364852e-11
>>     3 SNES Function norm 5.770723278635e+02 
>>       0 KSP preconditioned resid norm 1.348596653013e+02 true resid norm 5.770723278635e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.142365551700e-10 true resid norm 7.387664544933e-09 ||r(i)||/||b|| 1.280197331985e-11
>>     4 SNES Function norm 5.237702241066e+02 
>>       0 KSP preconditioned resid norm 1.524401223470e+02 true resid norm 5.237702241066e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.168592748079e-10 true resid norm 9.512653260875e-09 ||r(i)||/||b|| 1.816188248788e-11
>>     5 SNES Function norm 4.753909020302e+02 
>>       0 KSP preconditioned resid norm 1.864261890160e+02 true resid norm 4.753909020302e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.744912816503e-10 true resid norm 1.889384889937e-08 ||r(i)||/||b|| 3.974381675939e-11
>>     6 SNES Function norm 4.221784591168e+02 
>>       0 KSP preconditioned resid norm 2.550992140633e+02 true resid norm 4.221784591168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.697718851690e-10 true resid norm 1.666255394677e-08 ||r(i)||/||b|| 3.946803439861e-11
>>     7 SNES Function norm 3.806525080909e+02 
>>       0 KSP preconditioned resid norm 5.183605288372e+02 true resid norm 3.806525080909e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.240268561213e-09 true resid norm 1.041310587019e-07 ||r(i)||/||b|| 2.735593657958e-10
>>     8 SNES Function norm 3.762054656363e+02 
>>       0 KSP preconditioned resid norm 5.320776249886e+03 true resid norm 3.762054656363e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.328856456273e-09 true resid norm 7.860313076537e-08 ||r(i)||/||b|| 2.089367059897e-10
>>     9 SNES Function norm 3.758975227207e+02 
>>       0 KSP preconditioned resid norm 6.252711400154e+03 true resid norm 3.758975227207e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.900963654627e-09 true resid norm 1.347299589292e-07 ||r(i)||/||b|| 3.584220453330e-10
>>    10 SNES Function norm 3.757032042008e+02 
>>       0 KSP preconditioned resid norm 5.890497230519e+02 true resid norm 3.757032042008e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.497822807230e-09 true resid norm 3.605311757144e-08 ||r(i)||/||b|| 9.596169840535e-11
>>    11 SNES Function norm 3.728798164183e+02 
>>       0 KSP preconditioned resid norm 1.081356468099e+03 true resid norm 3.728798164183e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.713971332514e-09 true resid norm 3.705072372867e-08 ||r(i)||/||b|| 9.936371478768e-11
>>    12 SNES Function norm 3.723078736486e+02 
>>       0 KSP preconditioned resid norm 3.745123875883e+03 true resid norm 3.723078736486e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.885886422519e-08 true resid norm 2.837206039485e-07 ||r(i)||/||b|| 7.620591022374e-10
>>    13 SNES Function norm 3.721848051981e+02 
>>       0 KSP preconditioned resid norm 3.495480420016e+03 true resid norm 3.721848051981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.874759607493e-08 true resid norm 2.751735241072e-07 ||r(i)||/||b|| 7.393464759014e-10
>>    14 SNES Function norm 3.720227571463e+02 
>>       0 KSP preconditioned resid norm 1.265998097646e+04 true resid norm 3.720227571463e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.125038941589e-08 true resid norm 5.141611324046e-07 ||r(i)||/||b|| 1.382069033488e-09
>>    15 SNES Function norm 3.720051993666e+02 
>>       0 KSP preconditioned resid norm 2.912369751490e+03 true resid norm 3.720051993666e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.170907138699e-09 true resid norm 3.623872061163e-08 ||r(i)||/||b|| 9.741455407971e-11
>>    16 SNES Function norm 3.718945615434e+02 
>>       0 KSP preconditioned resid norm 1.082053028195e+03 true resid norm 3.718945615434e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.660982704773e-09 true resid norm 9.903133228018e-08 ||r(i)||/||b|| 2.662887348210e-10
>>    17 SNES Function norm 3.700412140239e+02 
>>       0 KSP preconditioned resid norm 3.676306553630e+01 true resid norm 3.700412140239e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.389666775474e-10 true resid norm 4.384613797840e-10 ||r(i)||/||b|| 1.184898771183e-12
>>    18 SNES Function norm 3.351961239413e+02 
>>       0 KSP preconditioned resid norm 1.071519127966e+02 true resid norm 3.351961239413e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.391049839274e-11 true resid norm 1.771280721778e-09 ||r(i)||/||b|| 5.284311468017e-12
>>    19 SNES Function norm 3.096040290055e+02 
>>       0 KSP preconditioned resid norm 3.362840348158e+02 true resid norm 3.096040290055e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.847034649159e-10 true resid norm 3.697317355249e-09 ||r(i)||/||b|| 1.194208411023e-11
>>    20 SNES Function norm 3.008687462984e+02 
>>       0 KSP preconditioned resid norm 7.161404453567e+01 true resid norm 3.008687462984e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.352839847013e-11 true resid norm 7.837384020140e-10 ||r(i)||/||b|| 2.604917963917e-12
>>    21 SNES Function norm 2.752608806757e+02 
>>       0 KSP preconditioned resid norm 1.727563200247e+02 true resid norm 2.752608806757e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.663026698374e-10 true resid norm 5.807801532466e-09 ||r(i)||/||b|| 2.109926233692e-11
>>    22 SNES Function norm 2.707559177016e+02 
>>       0 KSP preconditioned resid norm 4.125046548626e+02 true resid norm 2.707559177016e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.109007703208e-10 true resid norm 5.875389803064e-09 ||r(i)||/||b|| 2.169994972940e-11
>>    23 SNES Function norm 2.698855074973e+02 
>>       0 KSP preconditioned resid norm 7.414116220947e+01 true resid norm 2.698855074973e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.222534270039e-11 true resid norm 6.928234778488e-10 ||r(i)||/||b|| 2.567101450810e-12
>>    24 SNES Function norm 2.616892369202e+02 
>>       0 KSP preconditioned resid norm 1.819161466540e+02 true resid norm 2.616892369202e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.013662651542e-09 true resid norm 4.195842561304e-09 ||r(i)||/||b|| 1.603368411588e-11
>>    25 SNES Function norm 2.600248325147e+02 
>>       0 KSP preconditioned resid norm 1.265123744997e+02 true resid norm 2.600248325147e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.572744639399e-09 true resid norm 7.386966838391e-09 ||r(i)||/||b|| 2.840869761150e-11
>>    26 SNES Function norm 2.593065715720e+02 
>>       0 KSP preconditioned resid norm 1.314965938655e+02 true resid norm 2.593065715720e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.156889834113e-10 true resid norm 3.157778817609e-09 ||r(i)||/||b|| 1.217778168314e-11
>>    27 SNES Function norm 2.517348459439e+02 
>>       0 KSP preconditioned resid norm 2.618005944700e+01 true resid norm 2.517348459439e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.483924363849e-11 true resid norm 7.418968110702e-10 ||r(i)||/||b|| 2.947135936976e-12
>>    28 SNES Function norm 2.298635352641e+02 
>>       0 KSP preconditioned resid norm 8.770042104744e+01 true resid norm 2.298635352641e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.098440763657e-09 true resid norm 3.936819814733e-09 ||r(i)||/||b|| 1.712676962968e-11
>>    29 SNES Function norm 2.269871118080e+02 
>>       0 KSP preconditioned resid norm 2.285655807501e+02 true resid norm 2.269871118080e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.216116049541e-08 true resid norm 9.327420966977e-08 ||r(i)||/||b|| 4.109229327024e-10
>>    30 SNES Function norm 2.252105956612e+02 
>>       0 KSP preconditioned resid norm 8.597156834231e+01 true resid norm 2.252105956612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.431615405953e-09 true resid norm 4.297181487549e-09 ||r(i)||/||b|| 1.908072519826e-11
>>    31 SNES Function norm 2.251883212710e+02 
>>       0 KSP preconditioned resid norm 7.066983917118e+01 true resid norm 2.251883212710e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.998573955346e-10 true resid norm 3.454505271056e-09 ||r(i)||/||b|| 1.534051700176e-11
>>    32 SNES Function norm 2.117895362999e+02 
>>       0 KSP preconditioned resid norm 3.830985324064e+02 true resid norm 2.117895362999e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.016573825110e-09 true resid norm 1.942657883557e-08 ||r(i)||/||b|| 9.172586698552e-11
>>    33 SNES Function norm 2.116404150058e+02 
>>       0 KSP preconditioned resid norm 5.329232125733e+02 true resid norm 2.116404150058e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.609537995048e-09 true resid norm 1.451505196923e-08 ||r(i)||/||b|| 6.858355465253e-11
>>    34 SNES Function norm 2.115566314991e+02 
>>       0 KSP preconditioned resid norm 6.609724236205e+02 true resid norm 2.115566314991e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.254105497370e-10 true resid norm 1.178528765891e-08 ||r(i)||/||b|| 5.570748397436e-11
>>    35 SNES Function norm 2.115028944726e+02 
>>       0 KSP preconditioned resid norm 8.217040556045e+02 true resid norm 2.115028944726e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.803324342385e-09 true resid norm 2.957366914143e-08 ||r(i)||/||b|| 1.398263093050e-10
>>    36 SNES Function norm 2.114773510911e+02 
>>       0 KSP preconditioned resid norm 1.091957913212e+03 true resid norm 2.114773510911e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.264170340401e-08 true resid norm 1.635833495717e-07 ||r(i)||/||b|| 7.735265678696e-10
>>    37 SNES Function norm 2.114573155596e+02 
>>       0 KSP preconditioned resid norm 1.295967660499e+03 true resid norm 2.114573155596e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.787403648651e-08 true resid norm 1.913077973453e-07 ||r(i)||/||b|| 9.047111793648e-10
>>    38 SNES Function norm 2.114434864633e+02 
>>       0 KSP preconditioned resid norm 1.567323979910e+03 true resid norm 2.114434864633e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.869044911030e-09 true resid norm 3.425540409553e-08 ||r(i)||/||b|| 1.620073744928e-10
>>    39 SNES Function norm 2.114337906450e+02 
>>       0 KSP preconditioned resid norm 1.835099219771e+03 true resid norm 2.114337906450e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.224665285731e-10 true resid norm 1.690151657152e-08 ||r(i)||/||b|| 7.993763210677e-11
>>    40 SNES Function norm 2.114267348757e+02 
>>       0 KSP preconditioned resid norm 2.146249211809e+03 true resid norm 2.114267348757e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.809299780773e-08 true resid norm 1.157600089098e-07 ||r(i)||/||b|| 5.475183116166e-10
>>    41 SNES Function norm 2.114215910347e+02 
>>       0 KSP preconditioned resid norm 2.510692990698e+03 true resid norm 2.114215910347e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.485943641401e-08 true resid norm 9.439839396991e-08 ||r(i)||/||b|| 4.464936315535e-10
>>    42 SNES Function norm 2.114206676373e+02 
>>       0 KSP preconditioned resid norm 3.401544811763e+03 true resid norm 2.114206676373e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.910138749741e-09 true resid norm 3.806926432018e-08 ||r(i)||/||b|| 1.800640625423e-10
>>    43 SNES Function norm 2.114186309697e+02 
>>       0 KSP preconditioned resid norm 3.955962524189e+03 true resid norm 2.114186309697e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.500201400572e-08 true resid norm 1.570197988244e-07 ||r(i)||/||b|| 7.426961290224e-10
>>    44 SNES Function norm 2.114171338084e+02 
>>       0 KSP preconditioned resid norm 4.636713819556e+03 true resid norm 2.114171338084e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.248004040828e-09 true resid norm 7.296845022145e-08 ||r(i)||/||b|| 3.451397193171e-10
>>    45 SNES Function norm 2.114170015592e+02 
>>       0 KSP preconditioned resid norm 6.321621979244e+03 true resid norm 2.114170015592e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.674310076597e-08 true resid norm 1.788726595879e-07 ||r(i)||/||b|| 8.460656345928e-10
>>    46 SNES Function norm 2.114164190412e+02 
>>       0 KSP preconditioned resid norm 7.414534654352e+03 true resid norm 2.114164190412e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.520165386851e-08 true resid norm 4.326674448851e-07 ||r(i)||/||b|| 2.046517705897e-09
>>    47 SNES Function norm 2.114162562951e+02 
>>       0 KSP preconditioned resid norm 9.810108530473e+03 true resid norm 2.114162562951e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.690981568660e-08 true resid norm 1.826808079437e-07 ||r(i)||/||b|| 8.640811787375e-10
>>    48 SNES Function norm 2.114160137794e+02 
>>       0 KSP preconditioned resid norm 1.136099973228e+04 true resid norm 2.114160137794e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.015775310105e-08 true resid norm 1.261980904678e-07 ||r(i)||/||b|| 5.969183138583e-10
>>    49 SNES Function norm 2.114158331092e+02 
>>       0 KSP preconditioned resid norm 1.315560026827e+04 true resid norm 2.114158331092e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.131165057283e-08 true resid norm 3.097361970470e-07 ||r(i)||/||b|| 1.465056767470e-09
>>    50 SNES Function norm 2.114158223286e+02 
>>       0 KSP preconditioned resid norm 1.794717411653e+04 true resid norm 2.114158223286e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.070352092082e-07 true resid norm 6.224398071454e-07 ||r(i)||/||b|| 2.944149592446e-09
>>    51 SNES Function norm 2.114157500757e+02 
>>       0 KSP preconditioned resid norm 2.077631649185e+04 true resid norm 2.114157500757e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.358541701826e-08 true resid norm 5.604873460299e-07 ||r(i)||/||b|| 2.651114431301e-09
>>    52 SNES Function norm 2.114156961871e+02 
>>       0 KSP preconditioned resid norm 2.405048247398e+04 true resid norm 2.114156961871e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.439555573169e-08 true resid norm 3.896791261159e-07 ||r(i)||/||b|| 1.843189191455e-09
>>    53 SNES Function norm 2.114156592651e+02 
>>       0 KSP preconditioned resid norm 2.914886710376e+04 true resid norm 2.114156592651e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.436687722935e-07 true resid norm 1.370484810325e-06 ||r(i)||/||b|| 6.482418639609e-09
>>    54 SNES Function norm 2.114156319126e+02 
>>       0 KSP preconditioned resid norm 3.373835363786e+04 true resid norm 2.114156319126e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.028533154395e-07 true resid norm 6.668879459515e-07 ||r(i)||/||b|| 3.154392794508e-09
>>    55 SNES Function norm 2.114156115019e+02 
>>       0 KSP preconditioned resid norm 3.904963797200e+04 true resid norm 2.114156115019e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.986152192528e-08 true resid norm 4.508900491313e-07 ||r(i)||/||b|| 2.132718799374e-09
>>    56 SNES Function norm 2.114155964342e+02 
>>       0 KSP preconditioned resid norm 4.594596937760e+04 true resid norm 2.114155964342e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.567731897537e-08 true resid norm 5.054454305445e-07 ||r(i)||/||b|| 2.390766996709e-09
>>    57 SNES Function norm 2.114155854345e+02 
>>       0 KSP preconditioned resid norm 5.317614782539e+04 true resid norm 2.114155854345e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.460625544912e-07 true resid norm 9.171801951021e-07 ||r(i)||/||b|| 4.338280894556e-09
>>    58 SNES Function norm 2.114155777135e+02 
>>       0 KSP preconditioned resid norm 6.400017583847e+04 true resid norm 2.114155777135e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.093685259962e-08 true resid norm 6.506435082837e-07 ||r(i)||/||b|| 3.077557081273e-09
>>    59 SNES Function norm 2.114155720468e+02 
>>       0 KSP preconditioned resid norm 7.406807436961e+04 true resid norm 2.114155720468e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.571128588743e-07 true resid norm 1.516640238741e-06 ||r(i)||/||b|| 7.173739493538e-09
>>    60 SNES Function norm 2.114155680388e+02 
>>       0 KSP preconditioned resid norm 8.892466432276e+04 true resid norm 2.114155680388e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.335222631846e-08 true resid norm 8.161361513986e-07 ||r(i)||/||b|| 3.860340839464e-09
>>    61 SNES Function norm 2.114155651044e+02 
>>       0 KSP preconditioned resid norm 1.029106627628e+05 true resid norm 2.114155651044e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.244108037270e-07 true resid norm 3.009616926997e-06 ||r(i)||/||b|| 1.423555037450e-08
>>    62 SNES Function norm 2.114155629136e+02 
>>       0 KSP preconditioned resid norm 1.190953464375e+05 true resid norm 2.114155629136e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 5.108162336627e-08 true resid norm 9.193387836453e-07 ||r(i)||/||b|| 4.348491525295e-09
>>    63 SNES Function norm 2.114155612779e+02 
>>       0 KSP preconditioned resid norm 1.378243866677e+05 true resid norm 2.114155612779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.138603793608e-07 true resid norm 2.482322226118e-06 ||r(i)||/||b|| 1.174143573497e-08
>>    64 SNES Function norm 2.114155602404e+02 
>>       0 KSP preconditioned resid norm 1.698559088929e+05 true resid norm 2.114155602404e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.608494949489e-08 true resid norm 1.125534374356e-06 ||r(i)||/||b|| 5.323801015765e-09
>>    65 SNES Function norm 2.114155594364e+02 
>>       0 KSP preconditioned resid norm 1.965648734846e+05 true resid norm 2.114155594364e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.689853079655e-07 true resid norm 2.539748843840e-06 ||r(i)||/||b|| 1.201306493529e-08
>>    66 SNES Function norm 2.114155588361e+02 
>>       0 KSP preconditioned resid norm 2.274726625349e+05 true resid norm 2.114155588361e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.937782777025e-07 true resid norm 5.126870538363e-06 ||r(i)||/||b|| 2.425020450996e-08
>>    67 SNES Function norm 2.114155583879e+02 
>>       0 KSP preconditioned resid norm 2.632394448333e+05 true resid norm 2.114155583879e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.229084794159e-07 true resid norm 3.026195587359e-06 ||r(i)||/||b|| 1.431396823599e-08
>>    68 SNES Function norm 2.114155581765e+02 
>>       0 KSP preconditioned resid norm 3.367643572924e+05 true resid norm 2.114155581765e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.632794900620e-08 true resid norm 2.446819364813e-06 ||r(i)||/||b|| 1.157350663271e-08
>>    69 SNES Function norm 2.114155579720e+02 
>>       0 KSP preconditioned resid norm 3.897127303190e+05 true resid norm 2.114155579720e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.933486330809e-06 true resid norm 2.730183810468e-05 ||r(i)||/||b|| 1.291382638372e-07
>>    70 SNES Function norm 2.114155578193e+02 
>>       0 KSP preconditioned resid norm 4.509848215072e+05 true resid norm 2.114155578193e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.706389710731e-08 true resid norm 3.785217910694e-06 ||r(i)||/||b|| 1.790415970205e-08
>>    71 SNES Function norm 2.114155577054e+02 
>>       0 KSP preconditioned resid norm 5.242625554614e+05 true resid norm 2.114155577054e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.321749665453e-06 true resid norm 9.069981074375e-06 ||r(i)||/||b|| 4.290119976419e-08
>>    72 SNES Function norm 2.114155576210e+02 
>>       0 KSP preconditioned resid norm 6.066871925199e+05 true resid norm 2.114155576210e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 3.327204488687e-07 true resid norm 7.090997425424e-06 ||r(i)||/||b|| 3.354056581842e-08
>>    73 SNES Function norm 2.114155575580e+02 
>>       0 KSP preconditioned resid norm 7.020695777561e+05 true resid norm 2.114155575580e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.123816847857e-06 true resid norm 8.125899637337e-06 ||r(i)||/||b|| 3.843567489166e-08
>>    74 SNES Function norm 2.114155575109e+02 
>>       0 KSP preconditioned resid norm 8.124474305522e+05 true resid norm 2.114155575109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.293550435673e-07 true resid norm 7.844829319667e-06 ||r(i)||/||b|| 3.710620643072e-08
>>    75 SNES Function norm 2.114155574865e+02 
>>       0 KSP preconditioned resid norm 1.029314742970e+06 true resid norm 2.114155574865e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.033392384072e-07 true resid norm 9.399925240813e-06 ||r(i)||/||b|| 4.446184260312e-08
>>    76 SNES Function norm 2.114155574646e+02 
>>       0 KSP preconditioned resid norm 1.191137352032e+06 true resid norm 2.114155574646e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 4.636526184597e-07 true resid norm 1.077765913224e-05 ||r(i)||/||b|| 5.097855267366e-08
>>    77 SNES Function norm 2.114155574575e+02 
>>       0 KSP preconditioned resid norm 1.563965951801e+06 true resid norm 2.114155574575e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.137216998536e-06 true resid norm 5.108848518118e-05 ||r(i)||/||b|| 2.416496013612e-07
>>    78 SNES Function norm 2.114155574558e+02 
>>       0 KSP preconditioned resid norm 2.108895892585e+06 true resid norm 2.114155574558e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.138935971807e-05 true resid norm 6.467637231732e-05 ||r(i)||/||b|| 3.059205911601e-07
>>    79 SNES Function norm 2.114155574511e+02 
>>       0 KSP preconditioned resid norm 2.564542138115e+06 true resid norm 2.114155574511e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.391129540980e-06 true resid norm 4.283623913226e-05 ||r(i)||/||b|| 2.026163052933e-07
>>    80 SNES Function norm 2.114155574475e+02 
>>       0 KSP preconditioned resid norm 2.967732587697e+06 true resid norm 2.114155574475e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 7.446508619620e-06 true resid norm 5.164837794317e-05 ||r(i)||/||b|| 2.442979058246e-07
>>    81 SNES Function norm 2.114155574449e+02 
>>       0 KSP preconditioned resid norm 3.434290246612e+06 true resid norm 2.114155574449e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.781979196562e-06 true resid norm 4.886329177979e-05 ||r(i)||/||b|| 2.311243901364e-07
>>    82 SNES Function norm 2.114155574429e+02 
>>       0 KSP preconditioned resid norm 3.974205574181e+06 true resid norm 2.114155574429e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.425678262963e-05 true resid norm 8.517034218173e-05 ||r(i)||/||b|| 4.028574964485e-07
>>    83 SNES Function norm 2.114155574416e+02 
>>       0 KSP preconditioned resid norm 4.785116089506e+06 true resid norm 2.114155574416e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.130667094145e-05 true resid norm 1.264608296603e-04 ||r(i)||/||b|| 5.981623641639e-07
>>    84 SNES Function norm 2.114155574406e+02 
>>       0 KSP preconditioned resid norm 5.537350742612e+06 true resid norm 2.114155574406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.299149833543e-06 true resid norm 5.419308321331e-05 ||r(i)||/||b|| 2.563344148812e-07
>>    85 SNES Function norm 2.114155574399e+02 
>>       0 KSP preconditioned resid norm 6.767161611151e+06 true resid norm 2.114155574399e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.781986133631e-05 true resid norm 1.232616969009e-04 ||r(i)||/||b|| 5.830303994346e-07
>>    86 SNES Function norm 2.114155574394e+02 
>>       0 KSP preconditioned resid norm 7.867884017146e+06 true resid norm 2.114155574394e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.002696664504e-05 true resid norm 1.398533205346e-04 ||r(i)||/||b|| 6.615091255747e-07
>>    87 SNES Function norm 2.114155574390e+02 
>>       0 KSP preconditioned resid norm 9.102289060164e+06 true resid norm 2.114155574390e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.231201322143e-05 true resid norm 1.368407286352e-04 ||r(i)||/||b|| 6.472595030036e-07
>>    88 SNES Function norm 2.114155574387e+02 
>>       0 KSP preconditioned resid norm 1.063720954841e+07 true resid norm 2.114155574387e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.517853677131e-05 true resid norm 1.273061147947e-04 ||r(i)||/||b|| 6.021605805032e-07
>>    89 SNES Function norm 2.114155574385e+02 
>>       0 KSP preconditioned resid norm 1.230866787650e+07 true resid norm 2.114155574385e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.745229646688e-05 true resid norm 1.389443348166e-04 ||r(i)||/||b|| 6.572096041560e-07
>>    90 SNES Function norm 2.114155574384e+02 
>>       0 KSP preconditioned resid norm 1.436772950622e+07 true resid norm 2.114155574384e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 9.955301916175e-05 true resid norm 5.633754040143e-04 ||r(i)||/||b|| 2.664777421494e-06
>>    91 SNES Function norm 2.114155574383e+02 
>>       0 KSP preconditioned resid norm 1.662010558966e+07 true resid norm 2.114155574383e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.147116976734e-05 true resid norm 1.351152564949e-04 ||r(i)||/||b|| 6.390979837628e-07
>>    92 SNES Function norm 2.114155574382e+02 
>>       0 KSP preconditioned resid norm 1.923255924422e+07 true resid norm 2.114155574382e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.030030646485e-05 true resid norm 1.877965090103e-04 ||r(i)||/||b|| 8.882814078868e-07
>>    93 SNES Function norm 2.114155574381e+02 
>>       0 KSP preconditioned resid norm 2.226793244995e+07 true resid norm 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 2.411985876631e-04 true resid norm 1.342893408929e-03 ||r(i)||/||b|| 6.351913857247e-06
>>    94 SNES Function norm 2.114155574381e+02 
>>       0 KSP preconditioned resid norm 2.576953439486e+07 true resid norm 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 6.147595096922e-05 true resid norm 4.578229507611e-04 ||r(i)||/||b|| 2.165512114193e-06
>>    95 SNES Function norm 2.114155574380e+02 
>>       0 KSP preconditioned resid norm 2.981662207354e+07 true resid norm 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 8.456246888386e-06 true resid norm 2.856742906943e-04 ||r(i)||/||b|| 1.351245358460e-06
>>    96 SNES Function norm 2.114155574380e+02 
>>       0 KSP preconditioned resid norm 3.451395888978e+07 true resid norm 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
>>       1 KSP preconditioned resid norm 1.972976681165e-05 true resid norm 3.126288732572e-04 ||r(i)||/||b|| 1.478741096662e-06
>>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 96
>> 
>> -gideon
>> 
>>> On Sep 8, 2015, at 10:31 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>> On Tue, Sep 8, 2015 at 9:19 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> Ok, going back and running the problem with the flags, -snes_fd -snes_mf_operator -ksp_monitor_true_residual -snes_max_funcs 500000, during the grid sequence, I still get:
>>> 
>>> Lets try using -pc_type lu instead of MUMPS, just to make sure. If we still
>>> get this, then the problem is really horribly conditioned without being singular.
>>> We are losing all precision for some reason. Is it possible that the two halves
>>> are so differently scaled that we make no progress on one with LU?
>>> 
>>>   Thanks,
>>> 
>>>      Matt
>>>  
>>>       0 SNES Function norm 9.131726454160e-09 
>>>         0 KSP preconditioned resid norm 4.201012306235e-09 true resid norm 9.131726454160e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>         1 KSP preconditioned resid norm 4.811278613811e-14 true resid norm 7.117495013042e-13 ||r(i)||/||b|| 7.794249037978e-05
>>>         2 KSP preconditioned resid norm 1.729322442321e-19 true resid norm 2.611605096203e-13 ||r(i)||/||b|| 2.859924800982e-05
>>>       1 SNES Function norm 2.066497600915e-12 
>>>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>     0 SNES Function norm 5.066222213176e+03 
>>>       0 KSP preconditioned resid norm 3.135051257976e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 3.425321207844e-04 true resid norm 1.654961474093e+00 ||r(i)||/||b|| 3.266657885218e-04
>>>       2 KSP preconditioned resid norm 2.739130995469e-09 true resid norm 7.943827483388e-03 ||r(i)||/||b|| 1.567998234015e-06
>>>     1 SNES Function norm 8.487402654330e+02 
>>>       0 KSP preconditioned resid norm 2.670376525819e+02 true resid norm 8.487402654330e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 5.518628784729e-02 true resid norm 1.820466086409e+00 ||r(i)||/||b|| 2.144903641964e-03
>>>       2 KSP preconditioned resid norm 1.680654269556e-07 true resid norm 1.635299027240e-01 ||r(i)||/||b|| 1.926736710678e-04
>>>     2 SNES Function norm 6.547660268557e+02 
>>>       0 KSP preconditioned resid norm 1.529984375428e+02 true resid norm 6.547660268557e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 3.106652647444e-02 true resid norm 9.350438798847e-01 ||r(i)||/||b|| 1.428058026124e-03
>>>       2 KSP preconditioned resid norm 1.739404692247e-07 true resid norm 6.642022841611e-02 ||r(i)||/||b|| 1.014411647701e-04
>>>     3 SNES Function norm 5.771035131885e+02 
>>>       0 KSP preconditioned resid norm 1.360697738776e+02 true resid norm 5.771035131885e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.323138791370e-03 true resid norm 6.419077863137e-02 ||r(i)||/||b|| 1.112292286642e-04
>>>     4 SNES Function norm 5.238679723156e+02 
>>>       0 KSP preconditioned resid norm 1.528156345475e+02 true resid norm 5.238679723156e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 4.115694692015e-02 true resid norm 1.230231531838e+00 ||r(i)||/||b|| 2.348361795054e-03
>>>       2 KSP preconditioned resid norm 5.444815701049e-08 true resid norm 6.640979326953e-02 ||r(i)||/||b|| 1.267681873660e-04
>>>     5 SNES Function norm 4.755828064476e+02 
>>>       0 KSP preconditioned resid norm 1.872910183528e+02 true resid norm 4.755828064476e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 2.182180755863e-02 true resid norm 5.918459188490e-01 ||r(i)||/||b|| 1.244464498769e-03
>>>       2 KSP preconditioned resid norm 7.426717097996e-08 true resid norm 8.473303784961e-02 ||r(i)||/||b|| 1.781667392111e-04
>>>     6 SNES Function norm 4.224257767287e+02 
>>>       0 KSP preconditioned resid norm 2.578229460575e+02 true resid norm 4.224257767287e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.258348180527e-02 true resid norm 3.257118054248e-01 ||r(i)||/||b|| 7.710509712432e-04
>>>       2 KSP preconditioned resid norm 1.122972268815e-07 true resid norm 1.274729638407e-01 ||r(i)||/||b|| 3.017641698568e-04
>>>     7 SNES Function norm 3.811223987907e+02 
>>>       0 KSP preconditioned resid norm 5.218684581634e+02 true resid norm 3.811223987907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 3.089005298384e-01 true resid norm 5.194667956652e+00 ||r(i)||/||b|| 1.362992039601e-02
>>>       2 KSP preconditioned resid norm 7.570453480881e-07 true resid norm 2.863356359719e-01 ||r(i)||/||b|| 7.512957435207e-04
>>>     8 SNES Function norm 3.771290745624e+02 
>>>       0 KSP preconditioned resid norm 5.827455360446e+03 true resid norm 3.771290745624e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 9.620225652579e-01 true resid norm 1.332007773441e+01 ||r(i)||/||b|| 3.531967867995e-02
>>>       2 KSP preconditioned resid norm 3.596801636560e-06 true resid norm 3.425465452374e+00 ||r(i)||/||b|| 9.083005483862e-03
>>>     9 SNES Function norm 3.767897776540e+02 
>>>       0 KSP preconditioned resid norm 5.185115189077e+03 true resid norm 3.767897776540e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 7.021105455600e-01 true resid norm 1.043813038739e+01 ||r(i)||/||b|| 2.770279611187e-02
>>>       2 KSP preconditioned resid norm 6.951879108628e-06 true resid norm 2.816562934830e+00 ||r(i)||/||b|| 7.475157506572e-03
>>>    10 SNES Function norm 3.746968158051e+02 
>>>       0 KSP preconditioned resid norm 8.165048723911e+02 true resid norm 3.746968158051e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 2.177528145230e-01 true resid norm 2.394044923481e+00 ||r(i)||/||b|| 6.389285476945e-03
>>>       2 KSP preconditioned resid norm 7.325770200602e-07 true resid norm 4.877299135281e-01 ||r(i)||/||b|| 1.301665487816e-03
>>>    11 SNES Function norm 3.726877791108e+02 
>>>       0 KSP preconditioned resid norm 5.024547551936e+02 true resid norm 3.726877791108e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.640017871568e-01 true resid norm 1.648101091752e+00 ||r(i)||/||b|| 4.422203206351e-03
>>>       2 KSP preconditioned resid norm 1.005738633834e-06 true resid norm 2.935508517682e-01 ||r(i)||/||b|| 7.876589151074e-04
>>>    12 SNES Function norm 3.668832918639e+02 
>>>       0 KSP preconditioned resid norm 7.762104953873e+02 true resid norm 3.668832918639e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 9.912259329681e-02 true resid norm 1.334600681831e+00 ||r(i)||/||b|| 3.637670919956e-03
>>>       2 KSP preconditioned resid norm 7.640320635302e-07 true resid norm 4.471334143387e-01 ||r(i)||/||b|| 1.218734742776e-03
>>>    13 SNES Function norm 3.631997889667e+02 
>>>       0 KSP preconditioned resid norm 2.515761606628e+02 true resid norm 3.631997889667e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 2.648297544435e-01 true resid norm 2.866637955219e+00 ||r(i)||/||b|| 7.892730233611e-03
>>>       2 KSP preconditioned resid norm 7.305731711999e-07 true resid norm 1.472589913546e-01 ||r(i)||/||b|| 4.054490003243e-04
>>>    14 SNES Function norm 3.557844076846e+02 
>>>       0 KSP preconditioned resid norm 6.995145168850e+02 true resid norm 3.557844076846e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 3.626527197886e-02 true resid norm 5.642316830909e-01 ||r(i)||/||b|| 1.585880861848e-03
>>>       2 KSP preconditioned resid norm 1.040702263139e-06 true resid norm 4.073703976052e-01 ||r(i)||/||b|| 1.144992272866e-03
>>>    15 SNES Function norm 3.534147071188e+02 
>>>       0 KSP preconditioned resid norm 2.698507093385e+02 true resid norm 3.534147071188e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.592052963868e-01 true resid norm 1.332346399928e+00 ||r(i)||/||b|| 3.769923472595e-03
>>>       2 KSP preconditioned resid norm 5.021635239169e-07 true resid norm 1.573071613558e-01 ||r(i)||/||b|| 4.451064378114e-04
>>>    16 SNES Function norm 3.499900270010e+02 
>>>       0 KSP preconditioned resid norm 7.295084253442e+01 true resid norm 3.499900270010e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 4.040248007780e-02 true resid norm 1.427506717563e-01 ||r(i)||/||b|| 4.078706841436e-04
>>>       2 KSP preconditioned resid norm 1.246491224109e-07 true resid norm 4.074099442269e-02 ||r(i)||/||b|| 1.164061581177e-04
>>>    17 SNES Function norm 3.473017914657e+02 
>>>       0 KSP preconditioned resid norm 4.169372990718e+02 true resid norm 3.473017914657e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.140552389441e-01 true resid norm 1.893314948416e+00 ||r(i)||/||b|| 5.451497789361e-03
>>>       2 KSP preconditioned resid norm 7.751532205539e-07 true resid norm 2.494992549211e-01 ||r(i)||/||b|| 7.183932275966e-04
>>>    18 SNES Function norm 3.464416771335e+02 
>>>       0 KSP preconditioned resid norm 2.968698304201e+02 true resid norm 3.464416771335e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.683113115222e-02 true resid norm 2.501212385243e-01 ||r(i)||/||b|| 7.219721385540e-04
>>>       2 KSP preconditioned resid norm 5.274081741342e-07 true resid norm 1.712806350336e-01 ||r(i)||/||b|| 4.943996243489e-04
>>>    19 SNES Function norm 3.460747441983e+02 
>>>       0 KSP preconditioned resid norm 3.707198448214e+02 true resid norm 3.460747441983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 8.740664131652e-02 true resid norm 9.477346570578e-01 ||r(i)||/||b|| 2.738525919461e-03
>>>       2 KSP preconditioned resid norm 4.649501179917e-07 true resid norm 2.217443031583e-01 ||r(i)||/||b|| 6.407410736429e-04
>>>    20 SNES Function norm 3.455596398964e+02 
>>>       0 KSP preconditioned resid norm 5.000331867417e+02 true resid norm 3.455596398964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 7.821124737573e-02 true resid norm 7.651194558377e-01 ||r(i)||/||b|| 2.214145888296e-03
>>>       2 KSP preconditioned resid norm 5.030283955029e-07 true resid norm 2.918161448112e-01 ||r(i)||/||b|| 8.444740389784e-04
>>>    21 SNES Function norm 3.453222949611e+02 
>>>       0 KSP preconditioned resid norm 7.399233760260e+02 true resid norm 3.453222949611e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.030913689264e-01 true resid norm 8.930798988128e-01 ||r(i)||/||b|| 2.586221370136e-03
>>>       2 KSP preconditioned resid norm 1.497261432173e-06 true resid norm 4.385255355661e-01 ||r(i)||/||b|| 1.269902181136e-03
>>>    22 SNES Function norm 3.452850731057e+02 
>>>       0 KSP preconditioned resid norm 1.850020325701e+03 true resid norm 3.452850731057e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 8.670597549481e-01 true resid norm 6.205758169419e+00 ||r(i)||/||b|| 1.797285389027e-02
>>>       2 KSP preconditioned resid norm 4.122440586032e-06 true resid norm 1.109676174925e+00 ||r(i)||/||b|| 3.213797124053e-03
>>>    23 SNES Function norm 3.452722963704e+02 
>>>       0 KSP preconditioned resid norm 3.362446711623e+03 true resid norm 3.452722963704e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 2.181412468129e+00 true resid norm 1.564006484993e+01 ||r(i)||/||b|| 4.529776936738e-02
>>>       2 KSP preconditioned resid norm 1.165193864469e-05 true resid norm 1.869411988347e+00 ||r(i)||/||b|| 5.414312147250e-03
>>>    24 SNES Function norm 3.452679622612e+02 
>>>       0 KSP preconditioned resid norm 6.149140486077e+03 true resid norm 3.452679622612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 3.400761945257e-01 true resid norm 3.795886928810e+00 ||r(i)||/||b|| 1.099403171945e-02
>>>       2 KSP preconditioned resid norm 1.754461070240e-06 true resid norm 3.648059240080e+00 ||r(i)||/||b|| 1.056587821294e-02
>>>    25 SNES Function norm 3.452667776779e+02 
>>>       0 KSP preconditioned resid norm 1.101683177789e+04 true resid norm 3.452667776779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 3.317993935110e+00 true resid norm 2.454711939414e+01 ||r(i)||/||b|| 7.109609432809e-02
>>>       2 KSP preconditioned resid norm 3.457477291813e-05 true resid norm 6.807469339003e+00 ||r(i)||/||b|| 1.971654899665e-02
>>>    26 SNES Function norm 3.452666586474e+02 
>>>       0 KSP preconditioned resid norm 3.879190850784e+04 true resid norm 3.452666586474e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.739562350939e+01 true resid norm 1.221587956319e+02 ||r(i)||/||b|| 3.538099974971e-01
>>>       2 KSP preconditioned resid norm 7.455463420492e-05 true resid norm 3.080916578781e+01 ||r(i)||/||b|| 8.923295955800e-02
>>>    27 SNES Function norm 3.452666425264e+02 
>>>       0 KSP preconditioned resid norm 5.709318799171e+04 true resid norm 3.452666425264e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 2.292899448408e-01 true resid norm 2.801354737138e+01 ||r(i)||/||b|| 8.113597990933e-02
>>>    28 SNES Function norm 3.452666374187e+02 
>>>       0 KSP preconditioned resid norm 2.488913348294e+05 true resid norm 3.452666374187e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 9.691998534170e+02 true resid norm 7.244072115265e+03 ||r(i)||/||b|| 2.098109498625e+01
>>>       2 KSP preconditioned resid norm 2.503929513070e-03 true resid norm 2.928791518636e+03 ||r(i)||/||b|| 8.482694825461e+00
>>>    29 SNES Function norm 3.452666374177e+02 
>>>       0 KSP preconditioned resid norm 2.533150641403e+05 true resid norm 3.452666374177e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.817060595281e+02 true resid norm 1.333479168158e+03 ||r(i)||/||b|| 3.862172082803e+00
>>>       2 KSP preconditioned resid norm 8.930133492283e-04 true resid norm 5.412764197416e+02 ||r(i)||/||b|| 1.567705538507e+00
>>>    30 SNES Function norm 3.452666373128e+02 
>>>       0 KSP preconditioned resid norm 2.169409532487e+05 true resid norm 3.452666373128e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.325438416139e+02 true resid norm 9.441618139691e+02 ||r(i)||/||b|| 2.734587451940e+00
>>>       2 KSP preconditioned resid norm 8.796591247860e-04 true resid norm 2.978002325996e+02 ||r(i)||/||b|| 8.625224693509e-01
>>>    31 SNES Function norm 3.452666372449e+02 
>>>       0 KSP preconditioned resid norm 1.459475274582e+05 true resid norm 3.452666372449e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.978987062939e+02 true resid norm 1.411576222768e+03 ||r(i)||/||b|| 4.088365542735e+00
>>>       2 KSP preconditioned resid norm 1.235854958341e-03 true resid norm 3.579390387115e+02 ||r(i)||/||b|| 1.036703231936e+00
>>>    32 SNES Function norm 3.452666369928e+02 
>>>       0 KSP preconditioned resid norm 1.264234515603e+05 true resid norm 3.452666369928e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 2.676975931928e+01 true resid norm 1.855746501202e+02 ||r(i)||/||b|| 5.374821376792e-01
>>>       2 KSP preconditioned resid norm 9.286140031498e-05 true resid norm 3.495715551553e+01 ||r(i)||/||b|| 1.012468387331e-01
>>>    33 SNES Function norm 3.452666245901e+02 
>>>       0 KSP preconditioned resid norm 1.544591438108e+06 true resid norm 3.452666245901e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 4.501087626485e+01 true resid norm 3.233291920369e+02 ||r(i)||/||b|| 9.364623424597e-01
>>>       2 KSP preconditioned resid norm 1.867215855867e-04 true resid norm 7.114686634655e+01 ||r(i)||/||b|| 2.060635499623e-01
>>>    34 SNES Function norm 3.452666216061e+02 
>>>       0 KSP preconditioned resid norm 3.431570281248e+05 true resid norm 3.452666216061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 2.140176766342e+02 true resid norm 1.760366951993e+03 ||r(i)||/||b|| 5.098572644538e+00
>>>       2 KSP preconditioned resid norm 2.324744486242e-03 true resid norm 1.080605579953e+03 ||r(i)||/||b|| 3.129771348664e+00
>>>    35 SNES Function norm 3.452666215921e+02 
>>>       0 KSP preconditioned resid norm 3.734063885343e+05 true resid norm 3.452666215921e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 8.279219133674e+00 true resid norm 1.691481367261e+02 ||r(i)||/||b|| 4.899058470990e-01
>>>       2 KSP preconditioned resid norm 4.046466703016e-04 true resid norm 1.964786369469e+02 ||r(i)||/||b|| 5.690635139908e-01
>>>    36 SNES Function norm 3.452666212200e+02 
>>>       0 KSP preconditioned resid norm 7.952390829283e+05 true resid norm 3.452666212200e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 3.707607129733e+01 true resid norm 2.806640653978e+02 ||r(i)||/||b|| 8.128908158167e-01
>>>       2 KSP preconditioned resid norm 1.296391885225e-04 true resid norm 1.079623998255e+02 ||r(i)||/||b|| 3.126928384910e-01
>>>    37 SNES Function norm 3.452666199636e+02 
>>>       0 KSP preconditioned resid norm 9.358626316710e+05 true resid norm 3.452666199636e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 4.722696571742e+01 true resid norm 3.251492717238e+02 ||r(i)||/||b|| 9.417338744130e-01
>>>       2 KSP preconditioned resid norm 2.862108374145e-04 true resid norm 3.320104402606e+01 ||r(i)||/||b|| 9.616059620694e-02
>>>    38 SNES Function norm 3.452666133867e+02 
>>>       0 KSP preconditioned resid norm 8.365871848022e+04 true resid norm 3.452666133867e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 2.482543031390e+02 true resid norm 1.772891732407e+03 ||r(i)||/||b|| 5.134848443690e+00
>>>       2 KSP preconditioned resid norm 1.439398367387e-03 true resid norm 3.088902452162e+02 ||r(i)||/||b|| 8.946426710257e-01
>>>    39 SNES Function norm 3.452666132332e+02 
>>>       0 KSP preconditioned resid norm 8.960112658041e+04 true resid norm 3.452666132332e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 2.505891399150e+01 true resid norm 1.787713293837e+02 ||r(i)||/||b|| 5.177776319280e-01
>>>       2 KSP preconditioned resid norm 6.384856870578e-05 true resid norm 2.543851208517e+01 ||r(i)||/||b|| 7.367787996341e-02
>>>    40 SNES Function norm 3.452665904125e+02 
>>>       0 KSP preconditioned resid norm 7.796808144307e+05 true resid norm 3.452665904125e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 9.020764189760e+01 true resid norm 6.961545395677e+02 ||r(i)||/||b|| 2.016281212544e+00
>>>       2 KSP preconditioned resid norm 3.506849883654e-04 true resid norm 4.956460306859e+02 ||r(i)||/||b|| 1.435545878024e+00
>>>    41 SNES Function norm 3.452665902941e+02 
>>>       0 KSP preconditioned resid norm 4.990797566798e+05 true resid norm 3.452665902941e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.012324354359e+02 true resid norm 7.390530765626e+02 ||r(i)||/||b|| 2.140528789458e+00
>>>       2 KSP preconditioned resid norm 7.111862680292e-04 true resid norm 4.066260265930e+02 ||r(i)||/||b|| 1.177716112777e+00
>>>    42 SNES Function norm 3.452665901159e+02 
>>>       0 KSP preconditioned resid norm 3.472654025442e+05 true resid norm 3.452665901159e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.552680791160e+02 true resid norm 1.064105325830e+03 ||r(i)||/||b|| 3.081981739016e+00
>>>       2 KSP preconditioned resid norm 2.695346315896e-03 true resid norm 5.359619569435e+02 ||r(i)||/||b|| 1.552313407340e+00
>>>    43 SNES Function norm 3.452665900083e+02 
>>>       0 KSP preconditioned resid norm 2.823873507449e+05 true resid norm 3.452665900083e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 4.616335635707e+00 true resid norm 1.564312339233e+02 ||r(i)||/||b|| 4.530737651723e-01
>>>       2 KSP preconditioned resid norm 1.580836053583e-04 true resid norm 1.777870114826e+02 ||r(i)||/||b|| 5.149267743465e-01
>>>    44 SNES Function norm 3.452665895210e+02 
>>>       0 KSP preconditioned resid norm 4.764215602672e+05 true resid norm 3.452665895210e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 6.491949190147e+01 true resid norm 4.652947485976e+02 ||r(i)||/||b|| 1.347639078670e+00
>>>       2 KSP preconditioned resid norm 2.266106888017e-04 true resid norm 1.316722015429e+02 ||r(i)||/||b|| 3.813638664707e-01
>>>    45 SNES Function norm 3.452665889075e+02 
>>>       0 KSP preconditioned resid norm 1.439659890301e+05 true resid norm 3.452665889075e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.431461947162e+01 true resid norm 1.403614551576e+02 ||r(i)||/||b|| 4.065306625865e-01
>>>       2 KSP preconditioned resid norm 4.711117728059e-04 true resid norm 1.095117712535e+02 ||r(i)||/||b|| 3.171803318704e-01
>>>    46 SNES Function norm 3.452665876547e+02 
>>>       0 KSP preconditioned resid norm 2.157313334766e+05 true resid norm 3.452665876547e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 2.967750174401e+00 true resid norm 1.053591658730e+02 ||r(i)||/||b|| 3.051530893525e-01
>>>       2 KSP preconditioned resid norm 4.512059841816e-04 true resid norm 1.189061020519e+02 ||r(i)||/||b|| 3.443892525473e-01
>>>    47 SNES Function norm 3.452665866229e+02 
>>>       0 KSP preconditioned resid norm 3.981118078251e+05 true resid norm 3.452665866229e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 7.013119377232e+01 true resid norm 4.969789311062e+02 ||r(i)||/||b|| 1.439406390196e+00
>>>       2 KSP preconditioned resid norm 1.872910444347e-04 true resid norm 1.396396719776e+02 ||r(i)||/||b|| 4.044401554852e-01
>>>    48 SNES Function norm 3.452665850954e+02 
>>>       0 KSP preconditioned resid norm 1.967554630062e+05 true resid norm 3.452665850954e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 6.006970327024e+00 true resid norm 9.616098906643e+01 ||r(i)||/||b|| 2.785122951874e-01
>>>       2 KSP preconditioned resid norm 3.184368801558e-05 true resid norm 9.951979756268e+01 ||r(i)||/||b|| 2.882404549377e-01
>>>    49 SNES Function norm 3.452665835600e+02 
>>>       0 KSP preconditioned resid norm 3.965648247310e+05 true resid norm 3.452665835600e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.862969730098e+01 true resid norm 1.790616593203e+02 ||r(i)||/||b|| 5.186185627174e-01
>>>       2 KSP preconditioned resid norm 2.964457441143e-04 true resid norm 1.447818294544e+02 ||r(i)||/||b|| 4.193334552146e-01
>>>    50 SNES Function norm 3.452665828612e+02 
>>>       0 KSP preconditioned resid norm 1.265022286683e+06 true resid norm 3.452665828612e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 4.401066920177e+01 true resid norm 3.143671803790e+02 ||r(i)||/||b|| 9.105056671685e-01
>>>       2 KSP preconditioned resid norm 1.444898193175e-04 true resid norm 7.578180059616e+01 ||r(i)||/||b|| 2.194877939480e-01
>>>    51 SNES Function norm 3.452665817118e+02 
>>>       0 KSP preconditioned resid norm 2.063334812849e+05 true resid norm 3.452665817118e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 1.074628697499e+01 true resid norm 1.155569679685e+02 ||r(i)||/||b|| 3.346891187543e-01
>>>       2 KSP preconditioned resid norm 1.326420665890e-04 true resid norm 9.261090091148e+01 ||r(i)||/||b|| 2.682301323584e-01
>>>    52 SNES Function norm 3.452665807811e+02 
>>>       0 KSP preconditioned resid norm 6.549963238965e+06 true resid norm 3.452665807811e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>       1 KSP preconditioned resid norm 5.246352248726e+01 true resid norm 3.604213542889e+02 ||r(i)||/||b|| 1.043892963731e+00
>>>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 52
>>> 
>>> which is a little different, but not so much different, than what i get if I use my Jacobian.
>>> 
>>> -gideon
>>> 
>>>> On Sep 8, 2015, at 10:00 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> Yup, now it?s working.
>>>> -gideon
>>>> 
>>>>> On Sep 8, 2015, at 9:58 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>> 
>>>>> On Tue, Sep 8, 2015 at 8:51 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> Going back to this point, a case that previously worked, is now giving me problems that I don?t understand:
>>>>> 
>>>>>   0 SNES Function norm 1.421454390131e-02 
>>>>>     0 KSP preconditioned resid norm 7.635339584553e-03 true resid norm 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>     1 KSP preconditioned resid norm 2.838956718317e-07 true resid norm 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
>>>>>     2 KSP preconditioned resid norm 5.889748199467e-12 true resid norm 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
>>>>>   1 SNES Function norm 4.476139939604e-06 
>>>>>     0 KSP preconditioned resid norm 4.897102387844e-05 true resid norm 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
>>>>>     1 KSP preconditioned resid norm 1.133649519960e-09 true resid norm 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
>>>>>     2 KSP preconditioned resid norm 2.223328042462e-14 true resid norm 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
>>>>>   2 SNES Function norm 5.447761863330e-09 
>>>>>     0 KSP preconditioned resid norm 9.703890346186e-09 true resid norm 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>     1 KSP preconditioned resid norm 1.982949185218e-13 true resid norm 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
>>>>>     2 KSP preconditioned resid norm 7.045835741315e-18 true resid norm 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
>>>>> Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT iterations 2
>>>>> 
>>>>> Somehow the maximum number of function evaluations is being exceeded. Oh, are you using -snes_fd? This uses a function evaluation
>>>>> for each column, so you might have to up it > 10000. The convergence is fine.
>>>>> 
>>>>>   Thanks,
>>>>> 
>>>>>      Matt
>>>>>  
>>>>> -gideon
>>>>> 
>>>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>>   You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>>>>> 
>>>>>>  Barry
>>>>>> 
>>>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> I went through my code and nothing jumps out at me as clearly wrong.  Is there a way to dump to disk vectors and matrices that would be useful for the comparison?  I imagine I?d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>>>>> 
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong.  The one thing i was curious about is why it seems that there is only an issue during grid sequencing.  I?m not seeing that kind of behavior on the coarsest mesh.
>>>>>>>> 
>>>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>>>>> 
>>>>>>>>   Matt
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Alright, I?ll take another look at what has been implemented in my Jacobian file.  One thing I?m a bit unclear about is do I need these  -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>>>>> 
>>>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>>>>> 
>>>>>>>>> Gideon,
>>>>>>>>> 
>>>>>>>>>  In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value.   If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>>>>> 
>>>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -gideon
>>>>>>>>>> 
>>>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't catch all problems.
>>>>>>>>>>> 
>>>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges. 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Off the top of my head, I can?t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand.  And before I made that change to the Jacobian file, changing J for Jpre, if I ran with  -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>>>>> 
>>>>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>>>>> 
>>>>>>>>>>>>   0 SNES Function norm 3.857327250267e-09 
>>>>>>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>>>>   1 SNES Function norm 2.479697868412e-12 
>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 
>>>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>>>>> 
>>>>>>>>>>>> -gideon
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>  0 SNES Function norm 1.132185384796e-08 
>>>>>>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>>>>>  1 SNES Function norm 2.177599365111e-12 
>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 
>>>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre  see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 
>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> then 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other options you normally use
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm  Once the debugger starts up type cont and when it crashes type where or bt  Send all output
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator, 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown 
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try 
>>>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously discussed that output.  Here is a case where things fail.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 
>>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 
>>>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 
>>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 
>>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 
>>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 
>>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 
>>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 
>>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 
>>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 
>>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 
>>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 
>>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 
>>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 
>>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 
>>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 
>>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 
>>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 
>>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 
>>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 
>>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 
>>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 
>>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 
>>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 
>>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 
>>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 
>>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 
>>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 
>>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 
>>>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 
>>>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 
>>>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 
>>>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 
>>>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 
>>>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 
>>>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 
>>>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 
>>>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 
>>>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 
>>>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 
>>>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 
>>>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 
>>>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 
>>>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 
>>>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 
>>>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 
>>>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 
>>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 
>>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 
>>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 
>>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 
>>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 
>>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 
>>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 
>>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 
>>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 
>>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 
>>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 
>>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 
>>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 
>>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 
>>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 
>>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 
>>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 
>>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 
>>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 
>>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 
>>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 
>>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 
>>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 
>>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 
>>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 
>>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 
>>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 
>>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 
>>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
>>>>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
>>>>>>>>>>>>>>>>>>>>>>    type: seqaij
>>>>>>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>>>>      MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0 
>>>>>>>>>>>>>>>>>>>>>>        PAR (host participation):            1 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):                         1 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):                         0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace increase): 20 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):                           0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):                       0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):                         0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):                            0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):               0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be allocated locally):0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):               0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space basis):          0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or solution):          0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):                     -8 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential ordering):        1 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):                          0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in inv(A)):    0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve phase):    0 
>>>>>>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):                        0 
>>>>>>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):      0.01 
>>>>>>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement): 1.49012e-08 
>>>>>>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):      0 
>>>>>>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):         -1 
>>>>>>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0 
>>>>>>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the elimination after analysis): 
>>>>>>>>>>>>>>>>>>>>>>          [0] 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the assembly after factorization): 
>>>>>>>>>>>>>>>>>>>>>>          [0]  143924 
>>>>>>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the elimination after factorization): 
>>>>>>>>>>>>>>>>>>>>>>          [0]  1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>>        [0] 7 
>>>>>>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): 
>>>>>>>>>>>>>>>>>>>>>>          [0] 7 
>>>>>>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this processor after factorization): 
>>>>>>>>>>>>>>>>>>>>>>          [0] 15991 
>>>>>>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06 
>>>>>>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the assembly after factorization): 143924 
>>>>>>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06 
>>>>>>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the complete tree): 11 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete tree): 3996 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use after analysis): 6 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the matrix factors after factorization): 127890 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix after factorization): 11 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after factorization): 8 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after factorization): 0 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative refinement after solution): 0 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the factors): 255801 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 
>>>>>>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
>>>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code.  I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence.  Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint.  The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.  
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions  too.  Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one.  So I now need to go back an refine them.  I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>>>>    SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>>>>    etc
>>>>>>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a continuation problem that roughly looks like this 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 	solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that.  But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work: 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing  
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 


From bsmith at mcs.anl.gov  Tue Sep  8 22:50:39 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 Sep 2015 22:50:39 -0500
Subject: [petsc-users] Setting absolute tolerance on ksp solver
In-Reply-To: <CA+8mhyV4VU29WdMUSr7qXkd6gA+1h9g7u3UG4O8Jtx68pe05Xw@mail.gmail.com>
References: <CA+8mhyV4VU29WdMUSr7qXkd6gA+1h9g7u3UG4O8Jtx68pe05Xw@mail.gmail.com>
Message-ID: <95FB0D3D-A020-4B9A-B760-6F7A1C4AFC9A@mcs.anl.gov>


  Instead of setting rtol to default you should set it zero so it never gets triggered.

> On Sep 8, 2015, at 10:38 PM, Amal Sahai <amalsahai at gmail.com> wrote:
> 
> I have been setting the tolerance for a ksp solver using KSPSetTolerances. I would like to set my convergence criteria using only the absolute convergence parameter. I use the following command to set this up:
> 
> KSPSetTolerances( ksp, PETSC_DEFAULT, atol, PETSC_DEFAULT, maxits)
> 
> The problem is my ksp solver considers only the value of rtol while stopping the iterative solve. I tried changing the value of rtol to some other parameter instead of PETSC_DEFAULT (while making sure that atol is the more stringent condition requiring more iterations) but to no avail.
> 
> How can I fix this?
> 
> Regards
> Amal


From knepley at gmail.com  Wed Sep  9 06:40:57 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 9 Sep 2015 06:40:57 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <2C740D3B-AC9B-4BFA-B91F-4F715882B8D8@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<2C740D3B-AC9B-4BFA-B91F-4F715882B8D8@mcs.anl.gov>
Message-ID: <CAMYG4GnaC7xOGQkeU7QqEjpOXanRAUF-xpfcrqxCaHjAGaLD7g@mail.gmail.com>

On Tue, Sep 8, 2015 at 9:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   yeah but the crappy nonlinear convergence could easily be do to a bad
> Jacobian and 9 times out of 10 bad Newton convergence comes from a bad
> Jacobian.
>

Oh, I thought this was the -snes_fd run. I am getting confused. I think we
need to see -snes_view every time and
it should tell us what Jaociban was used.

   Matt


> > On Sep 8, 2015, at 9:53 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Tue, Sep 8, 2015 at 9:48 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> > Your comment about the different pieces scaling different made me think
> of something, and I made a slight modification.  It still fails, but the
> conditioning looks much better, no?
> >
> > This is great. Now its clear that the nonlinear convergence is crappy.
> >
> > Lets start with looking at the distribution of residuals. Take iterate
> 50,
> > which is as converged as you get and save the residual (VecView with
> > binary). Then we can load it up and play with it. I recommend
> >
> >   a) Norms for both halves
> >
> >   b) Plot of the big half
> >
> > What usually happens is that some small part is screwing up, but you
> > have to iterate the whole system, and finding the descent direction is
> > hard. If you select the problem variables, Newton will converge, and then
> > the big system will converge once the problem is removed. This is the
> classic
> > "coarse basis selection" problem in the nonlinear regime.
> >
> >    Matt
> >
> >   Thanks,
> >
> >     Matt
> >
> >
> >       0 SNES Function norm 4.027474333769e-09
> >         0 KSP preconditioned resid norm 9.390682304142e-09 true resid
> norm 4.027474333769e-09 ||r(i)||/||b|| 1.000000000000e+00
> >         1 KSP preconditioned resid norm 4.937459249116e-20 true resid
> norm 3.737772204700e-19 ||r(i)||/||b|| 9.280685350023e-11
> >       1 SNES Function norm 2.666655509899e-12
> >     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations
> 1
> >     0 SNES Function norm 5.066222213175e+03
> >       0 KSP preconditioned resid norm 3.135087050015e+01 true resid norm
> 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.220576053756e-11 true resid norm
> 1.206309587478e-07 ||r(i)||/||b|| 2.381083056997e-11
> >     1 SNES Function norm 8.484697184819e+02
> >       0 KSP preconditioned resid norm 2.668083368240e+02 true resid norm
> 8.484697184819e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 8.388166746665e-10 true resid norm
> 2.735732864000e-08 ||r(i)||/||b|| 3.224314085004e-11
> >     2 SNES Function norm 6.549559723821e+02
> >       0 KSP preconditioned resid norm 1.552154273063e+02 true resid norm
> 6.549559723821e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.740227244709e-10 true resid norm
> 1.800722435354e-08 ||r(i)||/||b|| 2.749379364852e-11
> >     3 SNES Function norm 5.770723278635e+02
> >       0 KSP preconditioned resid norm 1.348596653013e+02 true resid norm
> 5.770723278635e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.142365551700e-10 true resid norm
> 7.387664544933e-09 ||r(i)||/||b|| 1.280197331985e-11
> >     4 SNES Function norm 5.237702241066e+02
> >       0 KSP preconditioned resid norm 1.524401223470e+02 true resid norm
> 5.237702241066e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.168592748079e-10 true resid norm
> 9.512653260875e-09 ||r(i)||/||b|| 1.816188248788e-11
> >     5 SNES Function norm 4.753909020302e+02
> >       0 KSP preconditioned resid norm 1.864261890160e+02 true resid norm
> 4.753909020302e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.744912816503e-10 true resid norm
> 1.889384889937e-08 ||r(i)||/||b|| 3.974381675939e-11
> >     6 SNES Function norm 4.221784591168e+02
> >       0 KSP preconditioned resid norm 2.550992140633e+02 true resid norm
> 4.221784591168e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.697718851690e-10 true resid norm
> 1.666255394677e-08 ||r(i)||/||b|| 3.946803439861e-11
> >     7 SNES Function norm 3.806525080909e+02
> >       0 KSP preconditioned resid norm 5.183605288372e+02 true resid norm
> 3.806525080909e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.240268561213e-09 true resid norm
> 1.041310587019e-07 ||r(i)||/||b|| 2.735593657958e-10
> >     8 SNES Function norm 3.762054656363e+02
> >       0 KSP preconditioned resid norm 5.320776249886e+03 true resid norm
> 3.762054656363e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.328856456273e-09 true resid norm
> 7.860313076537e-08 ||r(i)||/||b|| 2.089367059897e-10
> >     9 SNES Function norm 3.758975227207e+02
> >       0 KSP preconditioned resid norm 6.252711400154e+03 true resid norm
> 3.758975227207e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.900963654627e-09 true resid norm
> 1.347299589292e-07 ||r(i)||/||b|| 3.584220453330e-10
> >    10 SNES Function norm 3.757032042008e+02
> >       0 KSP preconditioned resid norm 5.890497230519e+02 true resid norm
> 3.757032042008e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.497822807230e-09 true resid norm
> 3.605311757144e-08 ||r(i)||/||b|| 9.596169840535e-11
> >    11 SNES Function norm 3.728798164183e+02
> >       0 KSP preconditioned resid norm 1.081356468099e+03 true resid norm
> 3.728798164183e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.713971332514e-09 true resid norm
> 3.705072372867e-08 ||r(i)||/||b|| 9.936371478768e-11
> >    12 SNES Function norm 3.723078736486e+02
> >       0 KSP preconditioned resid norm 3.745123875883e+03 true resid norm
> 3.723078736486e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.885886422519e-08 true resid norm
> 2.837206039485e-07 ||r(i)||/||b|| 7.620591022374e-10
> >    13 SNES Function norm 3.721848051981e+02
> >       0 KSP preconditioned resid norm 3.495480420016e+03 true resid norm
> 3.721848051981e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.874759607493e-08 true resid norm
> 2.751735241072e-07 ||r(i)||/||b|| 7.393464759014e-10
> >    14 SNES Function norm 3.720227571463e+02
> >       0 KSP preconditioned resid norm 1.265998097646e+04 true resid norm
> 3.720227571463e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.125038941589e-08 true resid norm
> 5.141611324046e-07 ||r(i)||/||b|| 1.382069033488e-09
> >    15 SNES Function norm 3.720051993666e+02
> >       0 KSP preconditioned resid norm 2.912369751490e+03 true resid norm
> 3.720051993666e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.170907138699e-09 true resid norm
> 3.623872061163e-08 ||r(i)||/||b|| 9.741455407971e-11
> >    16 SNES Function norm 3.718945615434e+02
> >       0 KSP preconditioned resid norm 1.082053028195e+03 true resid norm
> 3.718945615434e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.660982704773e-09 true resid norm
> 9.903133228018e-08 ||r(i)||/||b|| 2.662887348210e-10
> >    17 SNES Function norm 3.700412140239e+02
> >       0 KSP preconditioned resid norm 3.676306553630e+01 true resid norm
> 3.700412140239e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.389666775474e-10 true resid norm
> 4.384613797840e-10 ||r(i)||/||b|| 1.184898771183e-12
> >    18 SNES Function norm 3.351961239413e+02
> >       0 KSP preconditioned resid norm 1.071519127966e+02 true resid norm
> 3.351961239413e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.391049839274e-11 true resid norm
> 1.771280721778e-09 ||r(i)||/||b|| 5.284311468017e-12
> >    19 SNES Function norm 3.096040290055e+02
> >       0 KSP preconditioned resid norm 3.362840348158e+02 true resid norm
> 3.096040290055e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.847034649159e-10 true resid norm
> 3.697317355249e-09 ||r(i)||/||b|| 1.194208411023e-11
> >    20 SNES Function norm 3.008687462984e+02
> >       0 KSP preconditioned resid norm 7.161404453567e+01 true resid norm
> 3.008687462984e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.352839847013e-11 true resid norm
> 7.837384020140e-10 ||r(i)||/||b|| 2.604917963917e-12
> >    21 SNES Function norm 2.752608806757e+02
> >       0 KSP preconditioned resid norm 1.727563200247e+02 true resid norm
> 2.752608806757e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.663026698374e-10 true resid norm
> 5.807801532466e-09 ||r(i)||/||b|| 2.109926233692e-11
> >    22 SNES Function norm 2.707559177016e+02
> >       0 KSP preconditioned resid norm 4.125046548626e+02 true resid norm
> 2.707559177016e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.109007703208e-10 true resid norm
> 5.875389803064e-09 ||r(i)||/||b|| 2.169994972940e-11
> >    23 SNES Function norm 2.698855074973e+02
> >       0 KSP preconditioned resid norm 7.414116220947e+01 true resid norm
> 2.698855074973e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.222534270039e-11 true resid norm
> 6.928234778488e-10 ||r(i)||/||b|| 2.567101450810e-12
> >    24 SNES Function norm 2.616892369202e+02
> >       0 KSP preconditioned resid norm 1.819161466540e+02 true resid norm
> 2.616892369202e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.013662651542e-09 true resid norm
> 4.195842561304e-09 ||r(i)||/||b|| 1.603368411588e-11
> >    25 SNES Function norm 2.600248325147e+02
> >       0 KSP preconditioned resid norm 1.265123744997e+02 true resid norm
> 2.600248325147e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.572744639399e-09 true resid norm
> 7.386966838391e-09 ||r(i)||/||b|| 2.840869761150e-11
> >    26 SNES Function norm 2.593065715720e+02
> >       0 KSP preconditioned resid norm 1.314965938655e+02 true resid norm
> 2.593065715720e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.156889834113e-10 true resid norm
> 3.157778817609e-09 ||r(i)||/||b|| 1.217778168314e-11
> >    27 SNES Function norm 2.517348459439e+02
> >       0 KSP preconditioned resid norm 2.618005944700e+01 true resid norm
> 2.517348459439e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 8.483924363849e-11 true resid norm
> 7.418968110702e-10 ||r(i)||/||b|| 2.947135936976e-12
> >    28 SNES Function norm 2.298635352641e+02
> >       0 KSP preconditioned resid norm 8.770042104744e+01 true resid norm
> 2.298635352641e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.098440763657e-09 true resid norm
> 3.936819814733e-09 ||r(i)||/||b|| 1.712676962968e-11
> >    29 SNES Function norm 2.269871118080e+02
> >       0 KSP preconditioned resid norm 2.285655807501e+02 true resid norm
> 2.269871118080e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.216116049541e-08 true resid norm
> 9.327420966977e-08 ||r(i)||/||b|| 4.109229327024e-10
> >    30 SNES Function norm 2.252105956612e+02
> >       0 KSP preconditioned resid norm 8.597156834231e+01 true resid norm
> 2.252105956612e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.431615405953e-09 true resid norm
> 4.297181487549e-09 ||r(i)||/||b|| 1.908072519826e-11
> >    31 SNES Function norm 2.251883212710e+02
> >       0 KSP preconditioned resid norm 7.066983917118e+01 true resid norm
> 2.251883212710e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.998573955346e-10 true resid norm
> 3.454505271056e-09 ||r(i)||/||b|| 1.534051700176e-11
> >    32 SNES Function norm 2.117895362999e+02
> >       0 KSP preconditioned resid norm 3.830985324064e+02 true resid norm
> 2.117895362999e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.016573825110e-09 true resid norm
> 1.942657883557e-08 ||r(i)||/||b|| 9.172586698552e-11
> >    33 SNES Function norm 2.116404150058e+02
> >       0 KSP preconditioned resid norm 5.329232125733e+02 true resid norm
> 2.116404150058e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.609537995048e-09 true resid norm
> 1.451505196923e-08 ||r(i)||/||b|| 6.858355465253e-11
> >    34 SNES Function norm 2.115566314991e+02
> >       0 KSP preconditioned resid norm 6.609724236205e+02 true resid norm
> 2.115566314991e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.254105497370e-10 true resid norm
> 1.178528765891e-08 ||r(i)||/||b|| 5.570748397436e-11
> >    35 SNES Function norm 2.115028944726e+02
> >       0 KSP preconditioned resid norm 8.217040556045e+02 true resid norm
> 2.115028944726e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.803324342385e-09 true resid norm
> 2.957366914143e-08 ||r(i)||/||b|| 1.398263093050e-10
> >    36 SNES Function norm 2.114773510911e+02
> >       0 KSP preconditioned resid norm 1.091957913212e+03 true resid norm
> 2.114773510911e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.264170340401e-08 true resid norm
> 1.635833495717e-07 ||r(i)||/||b|| 7.735265678696e-10
> >    37 SNES Function norm 2.114573155596e+02
> >       0 KSP preconditioned resid norm 1.295967660499e+03 true resid norm
> 2.114573155596e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.787403648651e-08 true resid norm
> 1.913077973453e-07 ||r(i)||/||b|| 9.047111793648e-10
> >    38 SNES Function norm 2.114434864633e+02
> >       0 KSP preconditioned resid norm 1.567323979910e+03 true resid norm
> 2.114434864633e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.869044911030e-09 true resid norm
> 3.425540409553e-08 ||r(i)||/||b|| 1.620073744928e-10
> >    39 SNES Function norm 2.114337906450e+02
> >       0 KSP preconditioned resid norm 1.835099219771e+03 true resid norm
> 2.114337906450e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.224665285731e-10 true resid norm
> 1.690151657152e-08 ||r(i)||/||b|| 7.993763210677e-11
> >    40 SNES Function norm 2.114267348757e+02
> >       0 KSP preconditioned resid norm 2.146249211809e+03 true resid norm
> 2.114267348757e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.809299780773e-08 true resid norm
> 1.157600089098e-07 ||r(i)||/||b|| 5.475183116166e-10
> >    41 SNES Function norm 2.114215910347e+02
> >       0 KSP preconditioned resid norm 2.510692990698e+03 true resid norm
> 2.114215910347e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.485943641401e-08 true resid norm
> 9.439839396991e-08 ||r(i)||/||b|| 4.464936315535e-10
> >    42 SNES Function norm 2.114206676373e+02
> >       0 KSP preconditioned resid norm 3.401544811763e+03 true resid norm
> 2.114206676373e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.910138749741e-09 true resid norm
> 3.806926432018e-08 ||r(i)||/||b|| 1.800640625423e-10
> >    43 SNES Function norm 2.114186309697e+02
> >       0 KSP preconditioned resid norm 3.955962524189e+03 true resid norm
> 2.114186309697e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.500201400572e-08 true resid norm
> 1.570197988244e-07 ||r(i)||/||b|| 7.426961290224e-10
> >    44 SNES Function norm 2.114171338084e+02
> >       0 KSP preconditioned resid norm 4.636713819556e+03 true resid norm
> 2.114171338084e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.248004040828e-09 true resid norm
> 7.296845022145e-08 ||r(i)||/||b|| 3.451397193171e-10
> >    45 SNES Function norm 2.114170015592e+02
> >       0 KSP preconditioned resid norm 6.321621979244e+03 true resid norm
> 2.114170015592e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.674310076597e-08 true resid norm
> 1.788726595879e-07 ||r(i)||/||b|| 8.460656345928e-10
> >    46 SNES Function norm 2.114164190412e+02
> >       0 KSP preconditioned resid norm 7.414534654352e+03 true resid norm
> 2.114164190412e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.520165386851e-08 true resid norm
> 4.326674448851e-07 ||r(i)||/||b|| 2.046517705897e-09
> >    47 SNES Function norm 2.114162562951e+02
> >       0 KSP preconditioned resid norm 9.810108530473e+03 true resid norm
> 2.114162562951e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.690981568660e-08 true resid norm
> 1.826808079437e-07 ||r(i)||/||b|| 8.640811787375e-10
> >    48 SNES Function norm 2.114160137794e+02
> >       0 KSP preconditioned resid norm 1.136099973228e+04 true resid norm
> 2.114160137794e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.015775310105e-08 true resid norm
> 1.261980904678e-07 ||r(i)||/||b|| 5.969183138583e-10
> >    49 SNES Function norm 2.114158331092e+02
> >       0 KSP preconditioned resid norm 1.315560026827e+04 true resid norm
> 2.114158331092e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.131165057283e-08 true resid norm
> 3.097361970470e-07 ||r(i)||/||b|| 1.465056767470e-09
> >    50 SNES Function norm 2.114158223286e+02
> >       0 KSP preconditioned resid norm 1.794717411653e+04 true resid norm
> 2.114158223286e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.070352092082e-07 true resid norm
> 6.224398071454e-07 ||r(i)||/||b|| 2.944149592446e-09
> >    51 SNES Function norm 2.114157500757e+02
> >       0 KSP preconditioned resid norm 2.077631649185e+04 true resid norm
> 2.114157500757e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.358541701826e-08 true resid norm
> 5.604873460299e-07 ||r(i)||/||b|| 2.651114431301e-09
> >    52 SNES Function norm 2.114156961871e+02
> >       0 KSP preconditioned resid norm 2.405048247398e+04 true resid norm
> 2.114156961871e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.439555573169e-08 true resid norm
> 3.896791261159e-07 ||r(i)||/||b|| 1.843189191455e-09
> >    53 SNES Function norm 2.114156592651e+02
> >       0 KSP preconditioned resid norm 2.914886710376e+04 true resid norm
> 2.114156592651e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.436687722935e-07 true resid norm
> 1.370484810325e-06 ||r(i)||/||b|| 6.482418639609e-09
> >    54 SNES Function norm 2.114156319126e+02
> >       0 KSP preconditioned resid norm 3.373835363786e+04 true resid norm
> 2.114156319126e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.028533154395e-07 true resid norm
> 6.668879459515e-07 ||r(i)||/||b|| 3.154392794508e-09
> >    55 SNES Function norm 2.114156115019e+02
> >       0 KSP preconditioned resid norm 3.904963797200e+04 true resid norm
> 2.114156115019e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.986152192528e-08 true resid norm
> 4.508900491313e-07 ||r(i)||/||b|| 2.132718799374e-09
> >    56 SNES Function norm 2.114155964342e+02
> >       0 KSP preconditioned resid norm 4.594596937760e+04 true resid norm
> 2.114155964342e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.567731897537e-08 true resid norm
> 5.054454305445e-07 ||r(i)||/||b|| 2.390766996709e-09
> >    57 SNES Function norm 2.114155854345e+02
> >       0 KSP preconditioned resid norm 5.317614782539e+04 true resid norm
> 2.114155854345e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.460625544912e-07 true resid norm
> 9.171801951021e-07 ||r(i)||/||b|| 4.338280894556e-09
> >    58 SNES Function norm 2.114155777135e+02
> >       0 KSP preconditioned resid norm 6.400017583847e+04 true resid norm
> 2.114155777135e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 8.093685259962e-08 true resid norm
> 6.506435082837e-07 ||r(i)||/||b|| 3.077557081273e-09
> >    59 SNES Function norm 2.114155720468e+02
> >       0 KSP preconditioned resid norm 7.406807436961e+04 true resid norm
> 2.114155720468e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.571128588743e-07 true resid norm
> 1.516640238741e-06 ||r(i)||/||b|| 7.173739493538e-09
> >    60 SNES Function norm 2.114155680388e+02
> >       0 KSP preconditioned resid norm 8.892466432276e+04 true resid norm
> 2.114155680388e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.335222631846e-08 true resid norm
> 8.161361513986e-07 ||r(i)||/||b|| 3.860340839464e-09
> >    61 SNES Function norm 2.114155651044e+02
> >       0 KSP preconditioned resid norm 1.029106627628e+05 true resid norm
> 2.114155651044e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.244108037270e-07 true resid norm
> 3.009616926997e-06 ||r(i)||/||b|| 1.423555037450e-08
> >    62 SNES Function norm 2.114155629136e+02
> >       0 KSP preconditioned resid norm 1.190953464375e+05 true resid norm
> 2.114155629136e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.108162336627e-08 true resid norm
> 9.193387836453e-07 ||r(i)||/||b|| 4.348491525295e-09
> >    63 SNES Function norm 2.114155612779e+02
> >       0 KSP preconditioned resid norm 1.378243866677e+05 true resid norm
> 2.114155612779e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.138603793608e-07 true resid norm
> 2.482322226118e-06 ||r(i)||/||b|| 1.174143573497e-08
> >    64 SNES Function norm 2.114155602404e+02
> >       0 KSP preconditioned resid norm 1.698559088929e+05 true resid norm
> 2.114155602404e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.608494949489e-08 true resid norm
> 1.125534374356e-06 ||r(i)||/||b|| 5.323801015765e-09
> >    65 SNES Function norm 2.114155594364e+02
> >       0 KSP preconditioned resid norm 1.965648734846e+05 true resid norm
> 2.114155594364e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.689853079655e-07 true resid norm
> 2.539748843840e-06 ||r(i)||/||b|| 1.201306493529e-08
> >    66 SNES Function norm 2.114155588361e+02
> >       0 KSP preconditioned resid norm 2.274726625349e+05 true resid norm
> 2.114155588361e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 8.937782777025e-07 true resid norm
> 5.126870538363e-06 ||r(i)||/||b|| 2.425020450996e-08
> >    67 SNES Function norm 2.114155583879e+02
> >       0 KSP preconditioned resid norm 2.632394448333e+05 true resid norm
> 2.114155583879e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.229084794159e-07 true resid norm
> 3.026195587359e-06 ||r(i)||/||b|| 1.431396823599e-08
> >    68 SNES Function norm 2.114155581765e+02
> >       0 KSP preconditioned resid norm 3.367643572924e+05 true resid norm
> 2.114155581765e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.632794900620e-08 true resid norm
> 2.446819364813e-06 ||r(i)||/||b|| 1.157350663271e-08
> >    69 SNES Function norm 2.114155579720e+02
> >       0 KSP preconditioned resid norm 3.897127303190e+05 true resid norm
> 2.114155579720e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.933486330809e-06 true resid norm
> 2.730183810468e-05 ||r(i)||/||b|| 1.291382638372e-07
> >    70 SNES Function norm 2.114155578193e+02
> >       0 KSP preconditioned resid norm 4.509848215072e+05 true resid norm
> 2.114155578193e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.706389710731e-08 true resid norm
> 3.785217910694e-06 ||r(i)||/||b|| 1.790415970205e-08
> >    71 SNES Function norm 2.114155577054e+02
> >       0 KSP preconditioned resid norm 5.242625554614e+05 true resid norm
> 2.114155577054e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.321749665453e-06 true resid norm
> 9.069981074375e-06 ||r(i)||/||b|| 4.290119976419e-08
> >    72 SNES Function norm 2.114155576210e+02
> >       0 KSP preconditioned resid norm 6.066871925199e+05 true resid norm
> 2.114155576210e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.327204488687e-07 true resid norm
> 7.090997425424e-06 ||r(i)||/||b|| 3.354056581842e-08
> >    73 SNES Function norm 2.114155575580e+02
> >       0 KSP preconditioned resid norm 7.020695777561e+05 true resid norm
> 2.114155575580e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.123816847857e-06 true resid norm
> 8.125899637337e-06 ||r(i)||/||b|| 3.843567489166e-08
> >    74 SNES Function norm 2.114155575109e+02
> >       0 KSP preconditioned resid norm 8.124474305522e+05 true resid norm
> 2.114155575109e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.293550435673e-07 true resid norm
> 7.844829319667e-06 ||r(i)||/||b|| 3.710620643072e-08
> >    75 SNES Function norm 2.114155574865e+02
> >       0 KSP preconditioned resid norm 1.029314742970e+06 true resid norm
> 2.114155574865e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.033392384072e-07 true resid norm
> 9.399925240813e-06 ||r(i)||/||b|| 4.446184260312e-08
> >    76 SNES Function norm 2.114155574646e+02
> >       0 KSP preconditioned resid norm 1.191137352032e+06 true resid norm
> 2.114155574646e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.636526184597e-07 true resid norm
> 1.077765913224e-05 ||r(i)||/||b|| 5.097855267366e-08
> >    77 SNES Function norm 2.114155574575e+02
> >       0 KSP preconditioned resid norm 1.563965951801e+06 true resid norm
> 2.114155574575e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.137216998536e-06 true resid norm
> 5.108848518118e-05 ||r(i)||/||b|| 2.416496013612e-07
> >    78 SNES Function norm 2.114155574558e+02
> >       0 KSP preconditioned resid norm 2.108895892585e+06 true resid norm
> 2.114155574558e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.138935971807e-05 true resid norm
> 6.467637231732e-05 ||r(i)||/||b|| 3.059205911601e-07
> >    79 SNES Function norm 2.114155574511e+02
> >       0 KSP preconditioned resid norm 2.564542138115e+06 true resid norm
> 2.114155574511e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.391129540980e-06 true resid norm
> 4.283623913226e-05 ||r(i)||/||b|| 2.026163052933e-07
> >    80 SNES Function norm 2.114155574475e+02
> >       0 KSP preconditioned resid norm 2.967732587697e+06 true resid norm
> 2.114155574475e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.446508619620e-06 true resid norm
> 5.164837794317e-05 ||r(i)||/||b|| 2.442979058246e-07
> >    81 SNES Function norm 2.114155574449e+02
> >       0 KSP preconditioned resid norm 3.434290246612e+06 true resid norm
> 2.114155574449e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.781979196562e-06 true resid norm
> 4.886329177979e-05 ||r(i)||/||b|| 2.311243901364e-07
> >    82 SNES Function norm 2.114155574429e+02
> >       0 KSP preconditioned resid norm 3.974205574181e+06 true resid norm
> 2.114155574429e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.425678262963e-05 true resid norm
> 8.517034218173e-05 ||r(i)||/||b|| 4.028574964485e-07
> >    83 SNES Function norm 2.114155574416e+02
> >       0 KSP preconditioned resid norm 4.785116089506e+06 true resid norm
> 2.114155574416e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.130667094145e-05 true resid norm
> 1.264608296603e-04 ||r(i)||/||b|| 5.981623641639e-07
> >    84 SNES Function norm 2.114155574406e+02
> >       0 KSP preconditioned resid norm 5.537350742612e+06 true resid norm
> 2.114155574406e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.299149833543e-06 true resid norm
> 5.419308321331e-05 ||r(i)||/||b|| 2.563344148812e-07
> >    85 SNES Function norm 2.114155574399e+02
> >       0 KSP preconditioned resid norm 6.767161611151e+06 true resid norm
> 2.114155574399e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.781986133631e-05 true resid norm
> 1.232616969009e-04 ||r(i)||/||b|| 5.830303994346e-07
> >    86 SNES Function norm 2.114155574394e+02
> >       0 KSP preconditioned resid norm 7.867884017146e+06 true resid norm
> 2.114155574394e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.002696664504e-05 true resid norm
> 1.398533205346e-04 ||r(i)||/||b|| 6.615091255747e-07
> >    87 SNES Function norm 2.114155574390e+02
> >       0 KSP preconditioned resid norm 9.102289060164e+06 true resid norm
> 2.114155574390e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.231201322143e-05 true resid norm
> 1.368407286352e-04 ||r(i)||/||b|| 6.472595030036e-07
> >    88 SNES Function norm 2.114155574387e+02
> >       0 KSP preconditioned resid norm 1.063720954841e+07 true resid norm
> 2.114155574387e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.517853677131e-05 true resid norm
> 1.273061147947e-04 ||r(i)||/||b|| 6.021605805032e-07
> >    89 SNES Function norm 2.114155574385e+02
> >       0 KSP preconditioned resid norm 1.230866787650e+07 true resid norm
> 2.114155574385e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.745229646688e-05 true resid norm
> 1.389443348166e-04 ||r(i)||/||b|| 6.572096041560e-07
> >    90 SNES Function norm 2.114155574384e+02
> >       0 KSP preconditioned resid norm 1.436772950622e+07 true resid norm
> 2.114155574384e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.955301916175e-05 true resid norm
> 5.633754040143e-04 ||r(i)||/||b|| 2.664777421494e-06
> >    91 SNES Function norm 2.114155574383e+02
> >       0 KSP preconditioned resid norm 1.662010558966e+07 true resid norm
> 2.114155574383e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.147116976734e-05 true resid norm
> 1.351152564949e-04 ||r(i)||/||b|| 6.390979837628e-07
> >    92 SNES Function norm 2.114155574382e+02
> >       0 KSP preconditioned resid norm 1.923255924422e+07 true resid norm
> 2.114155574382e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.030030646485e-05 true resid norm
> 1.877965090103e-04 ||r(i)||/||b|| 8.882814078868e-07
> >    93 SNES Function norm 2.114155574381e+02
> >       0 KSP preconditioned resid norm 2.226793244995e+07 true resid norm
> 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.411985876631e-04 true resid norm
> 1.342893408929e-03 ||r(i)||/||b|| 6.351913857247e-06
> >    94 SNES Function norm 2.114155574381e+02
> >       0 KSP preconditioned resid norm 2.576953439486e+07 true resid norm
> 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.147595096922e-05 true resid norm
> 4.578229507611e-04 ||r(i)||/||b|| 2.165512114193e-06
> >    95 SNES Function norm 2.114155574380e+02
> >       0 KSP preconditioned resid norm 2.981662207354e+07 true resid norm
> 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 8.456246888386e-06 true resid norm
> 2.856742906943e-04 ||r(i)||/||b|| 1.351245358460e-06
> >    96 SNES Function norm 2.114155574380e+02
> >       0 KSP preconditioned resid norm 3.451395888978e+07 true resid norm
> 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.972976681165e-05 true resid norm
> 3.126288732572e-04 ||r(i)||/||b|| 1.478741096662e-06
> >   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH
> iterations 96
> >
> > -gideon
> >
> >> On Sep 8, 2015, at 10:31 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >>
> >> On Tue, Sep 8, 2015 at 9:19 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >> Ok, going back and running the problem with the flags, -snes_fd
> -snes_mf_operator -ksp_monitor_true_residual -snes_max_funcs 500000, during
> the grid sequence, I still get:
> >>
> >> Lets try using -pc_type lu instead of MUMPS, just to make sure. If we
> still
> >> get this, then the problem is really horribly conditioned without being
> singular.
> >> We are losing all precision for some reason. Is it possible that the
> two halves
> >> are so differently scaled that we make no progress on one with LU?
> >>
> >>   Thanks,
> >>
> >>      Matt
> >>
> >>       0 SNES Function norm 9.131726454160e-09
> >>         0 KSP preconditioned resid norm 4.201012306235e-09 true resid
> norm 9.131726454160e-09 ||r(i)||/||b|| 1.000000000000e+00
> >>         1 KSP preconditioned resid norm 4.811278613811e-14 true resid
> norm 7.117495013042e-13 ||r(i)||/||b|| 7.794249037978e-05
> >>         2 KSP preconditioned resid norm 1.729322442321e-19 true resid
> norm 2.611605096203e-13 ||r(i)||/||b|| 2.859924800982e-05
> >>       1 SNES Function norm 2.066497600915e-12
> >>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE
> iterations 1
> >>     0 SNES Function norm 5.066222213176e+03
> >>       0 KSP preconditioned resid norm 3.135051257976e+01 true resid
> norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.425321207844e-04 true resid
> norm 1.654961474093e+00 ||r(i)||/||b|| 3.266657885218e-04
> >>       2 KSP preconditioned resid norm 2.739130995469e-09 true resid
> norm 7.943827483388e-03 ||r(i)||/||b|| 1.567998234015e-06
> >>     1 SNES Function norm 8.487402654330e+02
> >>       0 KSP preconditioned resid norm 2.670376525819e+02 true resid
> norm 8.487402654330e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 5.518628784729e-02 true resid
> norm 1.820466086409e+00 ||r(i)||/||b|| 2.144903641964e-03
> >>       2 KSP preconditioned resid norm 1.680654269556e-07 true resid
> norm 1.635299027240e-01 ||r(i)||/||b|| 1.926736710678e-04
> >>     2 SNES Function norm 6.547660268557e+02
> >>       0 KSP preconditioned resid norm 1.529984375428e+02 true resid
> norm 6.547660268557e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.106652647444e-02 true resid
> norm 9.350438798847e-01 ||r(i)||/||b|| 1.428058026124e-03
> >>       2 KSP preconditioned resid norm 1.739404692247e-07 true resid
> norm 6.642022841611e-02 ||r(i)||/||b|| 1.014411647701e-04
> >>     3 SNES Function norm 5.771035131885e+02
> >>       0 KSP preconditioned resid norm 1.360697738776e+02 true resid
> norm 5.771035131885e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.323138791370e-03 true resid
> norm 6.419077863137e-02 ||r(i)||/||b|| 1.112292286642e-04
> >>     4 SNES Function norm 5.238679723156e+02
> >>       0 KSP preconditioned resid norm 1.528156345475e+02 true resid
> norm 5.238679723156e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.115694692015e-02 true resid
> norm 1.230231531838e+00 ||r(i)||/||b|| 2.348361795054e-03
> >>       2 KSP preconditioned resid norm 5.444815701049e-08 true resid
> norm 6.640979326953e-02 ||r(i)||/||b|| 1.267681873660e-04
> >>     5 SNES Function norm 4.755828064476e+02
> >>       0 KSP preconditioned resid norm 1.872910183528e+02 true resid
> norm 4.755828064476e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.182180755863e-02 true resid
> norm 5.918459188490e-01 ||r(i)||/||b|| 1.244464498769e-03
> >>       2 KSP preconditioned resid norm 7.426717097996e-08 true resid
> norm 8.473303784961e-02 ||r(i)||/||b|| 1.781667392111e-04
> >>     6 SNES Function norm 4.224257767287e+02
> >>       0 KSP preconditioned resid norm 2.578229460575e+02 true resid
> norm 4.224257767287e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.258348180527e-02 true resid
> norm 3.257118054248e-01 ||r(i)||/||b|| 7.710509712432e-04
> >>       2 KSP preconditioned resid norm 1.122972268815e-07 true resid
> norm 1.274729638407e-01 ||r(i)||/||b|| 3.017641698568e-04
> >>     7 SNES Function norm 3.811223987907e+02
> >>       0 KSP preconditioned resid norm 5.218684581634e+02 true resid
> norm 3.811223987907e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.089005298384e-01 true resid
> norm 5.194667956652e+00 ||r(i)||/||b|| 1.362992039601e-02
> >>       2 KSP preconditioned resid norm 7.570453480881e-07 true resid
> norm 2.863356359719e-01 ||r(i)||/||b|| 7.512957435207e-04
> >>     8 SNES Function norm 3.771290745624e+02
> >>       0 KSP preconditioned resid norm 5.827455360446e+03 true resid
> norm 3.771290745624e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 9.620225652579e-01 true resid
> norm 1.332007773441e+01 ||r(i)||/||b|| 3.531967867995e-02
> >>       2 KSP preconditioned resid norm 3.596801636560e-06 true resid
> norm 3.425465452374e+00 ||r(i)||/||b|| 9.083005483862e-03
> >>     9 SNES Function norm 3.767897776540e+02
> >>       0 KSP preconditioned resid norm 5.185115189077e+03 true resid
> norm 3.767897776540e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 7.021105455600e-01 true resid
> norm 1.043813038739e+01 ||r(i)||/||b|| 2.770279611187e-02
> >>       2 KSP preconditioned resid norm 6.951879108628e-06 true resid
> norm 2.816562934830e+00 ||r(i)||/||b|| 7.475157506572e-03
> >>    10 SNES Function norm 3.746968158051e+02
> >>       0 KSP preconditioned resid norm 8.165048723911e+02 true resid
> norm 3.746968158051e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.177528145230e-01 true resid
> norm 2.394044923481e+00 ||r(i)||/||b|| 6.389285476945e-03
> >>       2 KSP preconditioned resid norm 7.325770200602e-07 true resid
> norm 4.877299135281e-01 ||r(i)||/||b|| 1.301665487816e-03
> >>    11 SNES Function norm 3.726877791108e+02
> >>       0 KSP preconditioned resid norm 5.024547551936e+02 true resid
> norm 3.726877791108e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.640017871568e-01 true resid
> norm 1.648101091752e+00 ||r(i)||/||b|| 4.422203206351e-03
> >>       2 KSP preconditioned resid norm 1.005738633834e-06 true resid
> norm 2.935508517682e-01 ||r(i)||/||b|| 7.876589151074e-04
> >>    12 SNES Function norm 3.668832918639e+02
> >>       0 KSP preconditioned resid norm 7.762104953873e+02 true resid
> norm 3.668832918639e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 9.912259329681e-02 true resid
> norm 1.334600681831e+00 ||r(i)||/||b|| 3.637670919956e-03
> >>       2 KSP preconditioned resid norm 7.640320635302e-07 true resid
> norm 4.471334143387e-01 ||r(i)||/||b|| 1.218734742776e-03
> >>    13 SNES Function norm 3.631997889667e+02
> >>       0 KSP preconditioned resid norm 2.515761606628e+02 true resid
> norm 3.631997889667e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.648297544435e-01 true resid
> norm 2.866637955219e+00 ||r(i)||/||b|| 7.892730233611e-03
> >>       2 KSP preconditioned resid norm 7.305731711999e-07 true resid
> norm 1.472589913546e-01 ||r(i)||/||b|| 4.054490003243e-04
> >>    14 SNES Function norm 3.557844076846e+02
> >>       0 KSP preconditioned resid norm 6.995145168850e+02 true resid
> norm 3.557844076846e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.626527197886e-02 true resid
> norm 5.642316830909e-01 ||r(i)||/||b|| 1.585880861848e-03
> >>       2 KSP preconditioned resid norm 1.040702263139e-06 true resid
> norm 4.073703976052e-01 ||r(i)||/||b|| 1.144992272866e-03
> >>    15 SNES Function norm 3.534147071188e+02
> >>       0 KSP preconditioned resid norm 2.698507093385e+02 true resid
> norm 3.534147071188e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.592052963868e-01 true resid
> norm 1.332346399928e+00 ||r(i)||/||b|| 3.769923472595e-03
> >>       2 KSP preconditioned resid norm 5.021635239169e-07 true resid
> norm 1.573071613558e-01 ||r(i)||/||b|| 4.451064378114e-04
> >>    16 SNES Function norm 3.499900270010e+02
> >>       0 KSP preconditioned resid norm 7.295084253442e+01 true resid
> norm 3.499900270010e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.040248007780e-02 true resid
> norm 1.427506717563e-01 ||r(i)||/||b|| 4.078706841436e-04
> >>       2 KSP preconditioned resid norm 1.246491224109e-07 true resid
> norm 4.074099442269e-02 ||r(i)||/||b|| 1.164061581177e-04
> >>    17 SNES Function norm 3.473017914657e+02
> >>       0 KSP preconditioned resid norm 4.169372990718e+02 true resid
> norm 3.473017914657e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.140552389441e-01 true resid
> norm 1.893314948416e+00 ||r(i)||/||b|| 5.451497789361e-03
> >>       2 KSP preconditioned resid norm 7.751532205539e-07 true resid
> norm 2.494992549211e-01 ||r(i)||/||b|| 7.183932275966e-04
> >>    18 SNES Function norm 3.464416771335e+02
> >>       0 KSP preconditioned resid norm 2.968698304201e+02 true resid
> norm 3.464416771335e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.683113115222e-02 true resid
> norm 2.501212385243e-01 ||r(i)||/||b|| 7.219721385540e-04
> >>       2 KSP preconditioned resid norm 5.274081741342e-07 true resid
> norm 1.712806350336e-01 ||r(i)||/||b|| 4.943996243489e-04
> >>    19 SNES Function norm 3.460747441983e+02
> >>       0 KSP preconditioned resid norm 3.707198448214e+02 true resid
> norm 3.460747441983e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 8.740664131652e-02 true resid
> norm 9.477346570578e-01 ||r(i)||/||b|| 2.738525919461e-03
> >>       2 KSP preconditioned resid norm 4.649501179917e-07 true resid
> norm 2.217443031583e-01 ||r(i)||/||b|| 6.407410736429e-04
> >>    20 SNES Function norm 3.455596398964e+02
> >>       0 KSP preconditioned resid norm 5.000331867417e+02 true resid
> norm 3.455596398964e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 7.821124737573e-02 true resid
> norm 7.651194558377e-01 ||r(i)||/||b|| 2.214145888296e-03
> >>       2 KSP preconditioned resid norm 5.030283955029e-07 true resid
> norm 2.918161448112e-01 ||r(i)||/||b|| 8.444740389784e-04
> >>    21 SNES Function norm 3.453222949611e+02
> >>       0 KSP preconditioned resid norm 7.399233760260e+02 true resid
> norm 3.453222949611e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.030913689264e-01 true resid
> norm 8.930798988128e-01 ||r(i)||/||b|| 2.586221370136e-03
> >>       2 KSP preconditioned resid norm 1.497261432173e-06 true resid
> norm 4.385255355661e-01 ||r(i)||/||b|| 1.269902181136e-03
> >>    22 SNES Function norm 3.452850731057e+02
> >>       0 KSP preconditioned resid norm 1.850020325701e+03 true resid
> norm 3.452850731057e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 8.670597549481e-01 true resid
> norm 6.205758169419e+00 ||r(i)||/||b|| 1.797285389027e-02
> >>       2 KSP preconditioned resid norm 4.122440586032e-06 true resid
> norm 1.109676174925e+00 ||r(i)||/||b|| 3.213797124053e-03
> >>    23 SNES Function norm 3.452722963704e+02
> >>       0 KSP preconditioned resid norm 3.362446711623e+03 true resid
> norm 3.452722963704e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.181412468129e+00 true resid
> norm 1.564006484993e+01 ||r(i)||/||b|| 4.529776936738e-02
> >>       2 KSP preconditioned resid norm 1.165193864469e-05 true resid
> norm 1.869411988347e+00 ||r(i)||/||b|| 5.414312147250e-03
> >>    24 SNES Function norm 3.452679622612e+02
> >>       0 KSP preconditioned resid norm 6.149140486077e+03 true resid
> norm 3.452679622612e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.400761945257e-01 true resid
> norm 3.795886928810e+00 ||r(i)||/||b|| 1.099403171945e-02
> >>       2 KSP preconditioned resid norm 1.754461070240e-06 true resid
> norm 3.648059240080e+00 ||r(i)||/||b|| 1.056587821294e-02
> >>    25 SNES Function norm 3.452667776779e+02
> >>       0 KSP preconditioned resid norm 1.101683177789e+04 true resid
> norm 3.452667776779e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.317993935110e+00 true resid
> norm 2.454711939414e+01 ||r(i)||/||b|| 7.109609432809e-02
> >>       2 KSP preconditioned resid norm 3.457477291813e-05 true resid
> norm 6.807469339003e+00 ||r(i)||/||b|| 1.971654899665e-02
> >>    26 SNES Function norm 3.452666586474e+02
> >>       0 KSP preconditioned resid norm 3.879190850784e+04 true resid
> norm 3.452666586474e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.739562350939e+01 true resid
> norm 1.221587956319e+02 ||r(i)||/||b|| 3.538099974971e-01
> >>       2 KSP preconditioned resid norm 7.455463420492e-05 true resid
> norm 3.080916578781e+01 ||r(i)||/||b|| 8.923295955800e-02
> >>    27 SNES Function norm 3.452666425264e+02
> >>       0 KSP preconditioned resid norm 5.709318799171e+04 true resid
> norm 3.452666425264e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.292899448408e-01 true resid
> norm 2.801354737138e+01 ||r(i)||/||b|| 8.113597990933e-02
> >>    28 SNES Function norm 3.452666374187e+02
> >>       0 KSP preconditioned resid norm 2.488913348294e+05 true resid
> norm 3.452666374187e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 9.691998534170e+02 true resid
> norm 7.244072115265e+03 ||r(i)||/||b|| 2.098109498625e+01
> >>       2 KSP preconditioned resid norm 2.503929513070e-03 true resid
> norm 2.928791518636e+03 ||r(i)||/||b|| 8.482694825461e+00
> >>    29 SNES Function norm 3.452666374177e+02
> >>       0 KSP preconditioned resid norm 2.533150641403e+05 true resid
> norm 3.452666374177e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.817060595281e+02 true resid
> norm 1.333479168158e+03 ||r(i)||/||b|| 3.862172082803e+00
> >>       2 KSP preconditioned resid norm 8.930133492283e-04 true resid
> norm 5.412764197416e+02 ||r(i)||/||b|| 1.567705538507e+00
> >>    30 SNES Function norm 3.452666373128e+02
> >>       0 KSP preconditioned resid norm 2.169409532487e+05 true resid
> norm 3.452666373128e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.325438416139e+02 true resid
> norm 9.441618139691e+02 ||r(i)||/||b|| 2.734587451940e+00
> >>       2 KSP preconditioned resid norm 8.796591247860e-04 true resid
> norm 2.978002325996e+02 ||r(i)||/||b|| 8.625224693509e-01
> >>    31 SNES Function norm 3.452666372449e+02
> >>       0 KSP preconditioned resid norm 1.459475274582e+05 true resid
> norm 3.452666372449e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.978987062939e+02 true resid
> norm 1.411576222768e+03 ||r(i)||/||b|| 4.088365542735e+00
> >>       2 KSP preconditioned resid norm 1.235854958341e-03 true resid
> norm 3.579390387115e+02 ||r(i)||/||b|| 1.036703231936e+00
> >>    32 SNES Function norm 3.452666369928e+02
> >>       0 KSP preconditioned resid norm 1.264234515603e+05 true resid
> norm 3.452666369928e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.676975931928e+01 true resid
> norm 1.855746501202e+02 ||r(i)||/||b|| 5.374821376792e-01
> >>       2 KSP preconditioned resid norm 9.286140031498e-05 true resid
> norm 3.495715551553e+01 ||r(i)||/||b|| 1.012468387331e-01
> >>    33 SNES Function norm 3.452666245901e+02
> >>       0 KSP preconditioned resid norm 1.544591438108e+06 true resid
> norm 3.452666245901e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.501087626485e+01 true resid
> norm 3.233291920369e+02 ||r(i)||/||b|| 9.364623424597e-01
> >>       2 KSP preconditioned resid norm 1.867215855867e-04 true resid
> norm 7.114686634655e+01 ||r(i)||/||b|| 2.060635499623e-01
> >>    34 SNES Function norm 3.452666216061e+02
> >>       0 KSP preconditioned resid norm 3.431570281248e+05 true resid
> norm 3.452666216061e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.140176766342e+02 true resid
> norm 1.760366951993e+03 ||r(i)||/||b|| 5.098572644538e+00
> >>       2 KSP preconditioned resid norm 2.324744486242e-03 true resid
> norm 1.080605579953e+03 ||r(i)||/||b|| 3.129771348664e+00
> >>    35 SNES Function norm 3.452666215921e+02
> >>       0 KSP preconditioned resid norm 3.734063885343e+05 true resid
> norm 3.452666215921e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 8.279219133674e+00 true resid
> norm 1.691481367261e+02 ||r(i)||/||b|| 4.899058470990e-01
> >>       2 KSP preconditioned resid norm 4.046466703016e-04 true resid
> norm 1.964786369469e+02 ||r(i)||/||b|| 5.690635139908e-01
> >>    36 SNES Function norm 3.452666212200e+02
> >>       0 KSP preconditioned resid norm 7.952390829283e+05 true resid
> norm 3.452666212200e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.707607129733e+01 true resid
> norm 2.806640653978e+02 ||r(i)||/||b|| 8.128908158167e-01
> >>       2 KSP preconditioned resid norm 1.296391885225e-04 true resid
> norm 1.079623998255e+02 ||r(i)||/||b|| 3.126928384910e-01
> >>    37 SNES Function norm 3.452666199636e+02
> >>       0 KSP preconditioned resid norm 9.358626316710e+05 true resid
> norm 3.452666199636e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.722696571742e+01 true resid
> norm 3.251492717238e+02 ||r(i)||/||b|| 9.417338744130e-01
> >>       2 KSP preconditioned resid norm 2.862108374145e-04 true resid
> norm 3.320104402606e+01 ||r(i)||/||b|| 9.616059620694e-02
> >>    38 SNES Function norm 3.452666133867e+02
> >>       0 KSP preconditioned resid norm 8.365871848022e+04 true resid
> norm 3.452666133867e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.482543031390e+02 true resid
> norm 1.772891732407e+03 ||r(i)||/||b|| 5.134848443690e+00
> >>       2 KSP preconditioned resid norm 1.439398367387e-03 true resid
> norm 3.088902452162e+02 ||r(i)||/||b|| 8.946426710257e-01
> >>    39 SNES Function norm 3.452666132332e+02
> >>       0 KSP preconditioned resid norm 8.960112658041e+04 true resid
> norm 3.452666132332e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.505891399150e+01 true resid
> norm 1.787713293837e+02 ||r(i)||/||b|| 5.177776319280e-01
> >>       2 KSP preconditioned resid norm 6.384856870578e-05 true resid
> norm 2.543851208517e+01 ||r(i)||/||b|| 7.367787996341e-02
> >>    40 SNES Function norm 3.452665904125e+02
> >>       0 KSP preconditioned resid norm 7.796808144307e+05 true resid
> norm 3.452665904125e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 9.020764189760e+01 true resid
> norm 6.961545395677e+02 ||r(i)||/||b|| 2.016281212544e+00
> >>       2 KSP preconditioned resid norm 3.506849883654e-04 true resid
> norm 4.956460306859e+02 ||r(i)||/||b|| 1.435545878024e+00
> >>    41 SNES Function norm 3.452665902941e+02
> >>       0 KSP preconditioned resid norm 4.990797566798e+05 true resid
> norm 3.452665902941e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.012324354359e+02 true resid
> norm 7.390530765626e+02 ||r(i)||/||b|| 2.140528789458e+00
> >>       2 KSP preconditioned resid norm 7.111862680292e-04 true resid
> norm 4.066260265930e+02 ||r(i)||/||b|| 1.177716112777e+00
> >>    42 SNES Function norm 3.452665901159e+02
> >>       0 KSP preconditioned resid norm 3.472654025442e+05 true resid
> norm 3.452665901159e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.552680791160e+02 true resid
> norm 1.064105325830e+03 ||r(i)||/||b|| 3.081981739016e+00
> >>       2 KSP preconditioned resid norm 2.695346315896e-03 true resid
> norm 5.359619569435e+02 ||r(i)||/||b|| 1.552313407340e+00
> >>    43 SNES Function norm 3.452665900083e+02
> >>       0 KSP preconditioned resid norm 2.823873507449e+05 true resid
> norm 3.452665900083e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.616335635707e+00 true resid
> norm 1.564312339233e+02 ||r(i)||/||b|| 4.530737651723e-01
> >>       2 KSP preconditioned resid norm 1.580836053583e-04 true resid
> norm 1.777870114826e+02 ||r(i)||/||b|| 5.149267743465e-01
> >>    44 SNES Function norm 3.452665895210e+02
> >>       0 KSP preconditioned resid norm 4.764215602672e+05 true resid
> norm 3.452665895210e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 6.491949190147e+01 true resid
> norm 4.652947485976e+02 ||r(i)||/||b|| 1.347639078670e+00
> >>       2 KSP preconditioned resid norm 2.266106888017e-04 true resid
> norm 1.316722015429e+02 ||r(i)||/||b|| 3.813638664707e-01
> >>    45 SNES Function norm 3.452665889075e+02
> >>       0 KSP preconditioned resid norm 1.439659890301e+05 true resid
> norm 3.452665889075e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.431461947162e+01 true resid
> norm 1.403614551576e+02 ||r(i)||/||b|| 4.065306625865e-01
> >>       2 KSP preconditioned resid norm 4.711117728059e-04 true resid
> norm 1.095117712535e+02 ||r(i)||/||b|| 3.171803318704e-01
> >>    46 SNES Function norm 3.452665876547e+02
> >>       0 KSP preconditioned resid norm 2.157313334766e+05 true resid
> norm 3.452665876547e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.967750174401e+00 true resid
> norm 1.053591658730e+02 ||r(i)||/||b|| 3.051530893525e-01
> >>       2 KSP preconditioned resid norm 4.512059841816e-04 true resid
> norm 1.189061020519e+02 ||r(i)||/||b|| 3.443892525473e-01
> >>    47 SNES Function norm 3.452665866229e+02
> >>       0 KSP preconditioned resid norm 3.981118078251e+05 true resid
> norm 3.452665866229e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 7.013119377232e+01 true resid
> norm 4.969789311062e+02 ||r(i)||/||b|| 1.439406390196e+00
> >>       2 KSP preconditioned resid norm 1.872910444347e-04 true resid
> norm 1.396396719776e+02 ||r(i)||/||b|| 4.044401554852e-01
> >>    48 SNES Function norm 3.452665850954e+02
> >>       0 KSP preconditioned resid norm 1.967554630062e+05 true resid
> norm 3.452665850954e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 6.006970327024e+00 true resid
> norm 9.616098906643e+01 ||r(i)||/||b|| 2.785122951874e-01
> >>       2 KSP preconditioned resid norm 3.184368801558e-05 true resid
> norm 9.951979756268e+01 ||r(i)||/||b|| 2.882404549377e-01
> >>    49 SNES Function norm 3.452665835600e+02
> >>       0 KSP preconditioned resid norm 3.965648247310e+05 true resid
> norm 3.452665835600e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.862969730098e+01 true resid
> norm 1.790616593203e+02 ||r(i)||/||b|| 5.186185627174e-01
> >>       2 KSP preconditioned resid norm 2.964457441143e-04 true resid
> norm 1.447818294544e+02 ||r(i)||/||b|| 4.193334552146e-01
> >>    50 SNES Function norm 3.452665828612e+02
> >>       0 KSP preconditioned resid norm 1.265022286683e+06 true resid
> norm 3.452665828612e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.401066920177e+01 true resid
> norm 3.143671803790e+02 ||r(i)||/||b|| 9.105056671685e-01
> >>       2 KSP preconditioned resid norm 1.444898193175e-04 true resid
> norm 7.578180059616e+01 ||r(i)||/||b|| 2.194877939480e-01
> >>    51 SNES Function norm 3.452665817118e+02
> >>       0 KSP preconditioned resid norm 2.063334812849e+05 true resid
> norm 3.452665817118e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.074628697499e+01 true resid
> norm 1.155569679685e+02 ||r(i)||/||b|| 3.346891187543e-01
> >>       2 KSP preconditioned resid norm 1.326420665890e-04 true resid
> norm 9.261090091148e+01 ||r(i)||/||b|| 2.682301323584e-01
> >>    52 SNES Function norm 3.452665807811e+02
> >>       0 KSP preconditioned resid norm 6.549963238965e+06 true resid
> norm 3.452665807811e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 5.246352248726e+01 true resid
> norm 3.604213542889e+02 ||r(i)||/||b|| 1.043892963731e+00
> >>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH
> iterations 52
> >>
> >> which is a little different, but not so much different, than what i get
> if I use my Jacobian.
> >>
> >> -gideon
> >>
> >>> On Sep 8, 2015, at 10:00 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> >>>
> >>> Yup, now it?s working.
> >>> -gideon
> >>>
> >>>> On Sep 8, 2015, at 9:58 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> >>>>
> >>>> On Tue, Sep 8, 2015 at 8:51 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>> Going back to this point, a case that previously worked, is now
> giving me problems that I don?t understand:
> >>>>
> >>>>   0 SNES Function norm 1.421454390131e-02
> >>>>     0 KSP preconditioned resid norm 7.635339584553e-03 true resid
> norm 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>     1 KSP preconditioned resid norm 2.838956718317e-07 true resid
> norm 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
> >>>>     2 KSP preconditioned resid norm 5.889748199467e-12 true resid
> norm 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
> >>>>   1 SNES Function norm 4.476139939604e-06
> >>>>     0 KSP preconditioned resid norm 4.897102387844e-05 true resid
> norm 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
> >>>>     1 KSP preconditioned resid norm 1.133649519960e-09 true resid
> norm 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
> >>>>     2 KSP preconditioned resid norm 2.223328042462e-14 true resid
> norm 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
> >>>>   2 SNES Function norm 5.447761863330e-09
> >>>>     0 KSP preconditioned resid norm 9.703890346186e-09 true resid
> norm 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
> >>>>     1 KSP preconditioned resid norm 1.982949185218e-13 true resid
> norm 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
> >>>>     2 KSP preconditioned resid norm 7.045835741315e-18 true resid
> norm 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
> >>>> Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT
> iterations 2
> >>>>
> >>>> Somehow the maximum number of function evaluations is being exceeded.
> Oh, are you using -snes_fd? This uses a function evaluation
> >>>> for each column, so you might have to up it > 10000. The convergence
> is fine.
> >>>>
> >>>>   Thanks,
> >>>>
> >>>>      Matt
> >>>>
> >>>> -gideon
> >>>>
> >>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >>>>>
> >>>>>
> >>>>>   You can run with -snes_fd it will be slow so run with only two
> levels of grid sequencing; check the convergence rate of Newton on the
> finer grid. It will hopefully be much better (also KSP should converge in
> one iteration). Send the output
> >>>>>
> >>>>>  Barry
> >>>>>
> >>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>
> >>>>>> I went through my code and nothing jumps out at me as clearly
> wrong.  Is there a way to dump to disk vectors and matrices that would be
> useful for the comparison?  I imagine I?d want the jacobian that I am
> specifying, but what would I want to compare that against in terms of using
> -snes_mf_operator?
> >>>>>>
> >>>>>>
> >>>>>> -gideon
> >>>>>>
> >>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>> Just to clarify, again, during the linear solve phase, even though
> the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that
> the true residual norm could be comparatively large signified there is
> something wrong.  The one thing i was curious about is why it seems that
> there is only an issue during grid sequencing.  I?m not seeing that kind of
> behavior on the coarsest mesh.
> >>>>>>>
> >>>>>>> That definitely sounds like a bug in the Jacobian evaluation.
> Maybe you assume a certain mesh size, and when
> >>>>>>> PETSc refines the DA and passes in the new one something goes
> wrong?
> >>>>>>>
> >>>>>>>   Matt
> >>>>>>>
> >>>>>>> -gideon
> >>>>>>>
> >>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Alright, I?ll take another look at what has been implemented in
> my Jacobian file.  One thing I?m a bit unclear about is do I need these
> -snes_mf and snes_mf_operator flags if I have, coded in,
> >>>>>>>>>
> >>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
> >>>>>>>>
> >>>>>>>> Gideon,
> >>>>>>>>
> >>>>>>>>  In your case the DM creates that Jacobian matrix object and
> form_function_jacobian's job is just to fill it up with the correct values
> for the current solution value.   If you use -snes_mf_operator then SNES
> uses a matrix-free multiple for the first matrix and your Jacobian to build
> the preconditioner. With -snes_mf it uses a matrix-free multiple for the
> first and second matrix and ignores what you compute. Since it is matrix
> free in both it cannot build a preconditioner hence the -pc_type none
> >>>>>>>>
> >>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the
> Jacobian you provide is correct
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Barry
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -gideon
> >>>>>>>>>
> >>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with
> a PC type of LU should give two iterations (or at most three) to get very
> good (10^-12) convergence of the linear system. Here you are not getting
> that after a few SNES iterations. So I am strongly leaning to something
> wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't
> catch all problems.
> >>>>>>>>>>
> >>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear
> solver converges.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Barry
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Off the top of my head, I can?t rule out a null space in this
> problem. I thought I did a pretty good job of coding the Jacobian by hand.
> And before I made that change to the Jacobian file, changing J for Jpre, if
> I ran with  -snes_check_jacobian, it appeared to be satisfactory.
> >>>>>>>>>>>
> >>>>>>>>>>> When I run with -ksp_type fgmres, I get
> >>>>>>>>>>>
> >>>>>>>>>>>   0 SNES Function norm 3.857327250267e-09
> >>>>>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true
> resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true
> resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
> >>>>>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true
> resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
> >>>>>>>>>>>   1 SNES Function norm 2.479697868412e-12
> >>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE
> iterations 1
> >>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true
> resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true
> resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
> >>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true
> resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true
> resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true
> resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
> >>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true
> resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true
> resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true
> resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
> >>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true
> resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true
> resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true
> resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
> >>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true
> resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true
> resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true
> resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
> >>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true
> resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true
> resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true
> resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
> >>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true
> resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true
> resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true
> resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
> >>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true
> resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true
> resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true
> resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
> >>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true
> resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true
> resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true
> resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true
> resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
> >>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true
> resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true
> resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true
> resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
> >>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true
> resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true
> resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true
> resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
> >>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true
> resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true
> resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true
> resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true
> resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
> >>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true
> resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true
> resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true
> resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
> >>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true
> resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true
> resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true
> resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
> >>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true
> resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true
> resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true
> resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
> >>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true
> resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true
> resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true
> resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
> >>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true
> resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true
> resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true
> resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
> >>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true
> resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true
> resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true
> resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true
> resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
> >>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true
> resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true
> resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true
> resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
> >>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true
> resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true
> resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true
> resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
> >>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true
> resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true
> resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true
> resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true
> resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
> >>>>>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true
> resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
> >>>>>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true
> resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
> >>>>>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true
> resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
> >>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true
> resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true
> resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true
> resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true
> resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
> >>>>>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true
> resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
> >>>>>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true
> resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
> >>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true
> resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
> >>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true
> resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true
> resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true
> resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true
> resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
> >>>>>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true
> resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
> >>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true
> resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true
> resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true
> resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true
> resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
> >>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true
> resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
> >>>>>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true
> resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
> >>>>>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true
> resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
> >>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true
> resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
> >>>>>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true
> resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
> >>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true
> resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true
> resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true
> resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true
> resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
> >>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true
> resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
> >>>>>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true
> resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
> >>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true
> resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
> >>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true
> resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
> >>>>>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true
> resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
> >>>>>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true
> resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
> >>>>>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true
> resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
> >>>>>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true
> resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
> >>>>>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true
> resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
> >>>>>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true
> resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
> >>>>>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true
> resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
> >>>>>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true
> resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
> >>>>>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true
> resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
> >>>>>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true
> resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
> >>>>>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true
> resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
> >>>>>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true
> resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
> >>>>>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true
> resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
> >>>>>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true
> resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
> >>>>>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true
> resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
> >>>>>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true
> resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
> >>>>>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true
> resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
> >>>>>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true
> resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
> >>>>>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true
> resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
> >>>>>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true
> resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
> >>>>>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true
> resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
> >>>>>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true
> resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
> >>>>>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true
> resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
> >>>>>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true
> resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
> >>>>>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true
> resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
> >>>>>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true
> resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
> >>>>>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true
> resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
> >>>>>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true
> resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
> >>>>>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true
> resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
> >>>>>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true
> resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
> >>>>>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true
> resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
> >>>>>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true
> resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
> >>>>>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true
> resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
> >>>>>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true
> resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
> >>>>>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true
> resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
> >>>>>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true
> resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
> >>>>>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true
> resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
> >>>>>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true
> resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
> >>>>>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true
> resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
> >>>>>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true
> resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
> >>>>>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true
> resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
> >>>>>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true
> resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
> >>>>>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true
> resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
> >>>>>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true
> resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
> >>>>>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true
> resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
> >>>>>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true
> resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
> >>>>>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true
> resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
> >>>>>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true
> resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
> >>>>>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true
> resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
> >>>>>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true
> resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
> >>>>>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true
> resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
> >>>>>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true
> resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
> >>>>>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true
> resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
> >>>>>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true
> resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
> >>>>>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true
> resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
> >>>>>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true
> resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
> >>>>>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true
> resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
> >>>>>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true
> resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
> >>>>>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true
> resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
> >>>>>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true
> resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
> >>>>>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true
> resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
> >>>>>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true
> resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
> >>>>>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true
> resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
> >>>>>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true
> resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
> >>>>>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true
> resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
> >>>>>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true
> resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
> >>>>>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true
> resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
> >>>>>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true
> resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
> >>>>>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true
> resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
> >>>>>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true
> resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
> >>>>>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true
> resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
> >>>>>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true
> resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
> >>>>>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true
> resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
> >>>>>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true
> resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
> >>>>>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true
> resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
> >>>>>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true
> resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
> >>>>>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true
> resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
> >>>>>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true
> resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
> >>>>>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true
> resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
> >>>>>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true
> resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
> >>>>>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true
> resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
> >>>>>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true
> resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
> >>>>>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true
> resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
> >>>>>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true
> resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
> >>>>>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true
> resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
> >>>>>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true
> resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
> >>>>>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true
> resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
> >>>>>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true
> resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
> >>>>>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true
> resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
> >>>>>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true
> resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
> >>>>>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true
> resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
> >>>>>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true
> resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
> >>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true
> resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
> >>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true
> resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
> >>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true
> resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
> >>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true
> resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
> >>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true
> resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
> >>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true
> resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
> >>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true
> resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
> >>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true
> resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
> >>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true
> resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
> >>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true
> resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
> >>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true
> resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
> >>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true
> resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
> >>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true
> resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
> >>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true
> resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
> >>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true
> resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
> >>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true
> resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
> >>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true
> resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
> >>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true
> resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
> >>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true
> resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
> >>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true
> resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
> >>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true
> resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
> >>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true
> resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
> >>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true
> resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
> >>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true
> resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
> >>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true
> resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
> >>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true
> resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
> >>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true
> resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
> >>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true
> resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
> >>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true
> resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
> >>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true
> resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
> >>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true
> resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
> >>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true
> resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
> >>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true
> resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
> >>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true
> resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
> >>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true
> resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
> >>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true
> resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
> >>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true
> resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
> >>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true
> resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
> >>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true
> resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
> >>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true
> resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
> >>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true
> resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
> >>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true
> resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
> >>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true
> resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
> >>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true
> resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
> >>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true
> resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
> >>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true
> resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
> >>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true
> resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
> >>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true
> resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
> >>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true
> resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
> >>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true
> resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
> >>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true
> resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
> >>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true
> resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
> >>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true
> resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
> >>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true
> resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
> >>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true
> resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
> >>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true
> resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
> >>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true
> resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
> >>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true
> resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
> >>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true
> resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
> >>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true
> resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
> >>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true
> resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
> >>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true
> resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
> >>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true
> resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
> >>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true
> resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
> >>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true
> resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
> >>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true
> resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
> >>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true
> resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
> >>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true
> resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
> >>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true
> resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
> >>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true
> resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
> >>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true
> resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
> >>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true
> resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
> >>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true
> resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
> >>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true
> resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
> >>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true
> resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
> >>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true
> resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
> >>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true
> resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
> >>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true
> resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
> >>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true
> resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
> >>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true
> resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
> >>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true
> resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
> >>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true
> resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
> >>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true
> resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
> >>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true
> resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
> >>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true
> resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
> >>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true
> resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
> >>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true
> resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
> >>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true
> resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
> >>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true
> resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
> >>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true
> resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
> >>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true
> resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
> >>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true
> resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
> >>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true
> resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
> >>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true
> resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
> >>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true
> resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
> >>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true
> resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
> >>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true
> resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
> >>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true
> resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
> >>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true
> resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
> >>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true
> resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
> >>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true
> resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
> >>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true
> resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
> >>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true
> resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
> >>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true
> resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
> >>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true
> resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
> >>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true
> resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
> >>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true
> resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
> >>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true
> resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
> >>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true
> resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
> >>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true
> resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
> >>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true
> resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
> >>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> iterations 24
> >>>>>>>>>>>
> >>>>>>>>>>> -gideon
> >>>>>>>>>>>
> >>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For
> example with Neumann boundary conditions there is often a nullspace)
> >>>>>>>>>>>>
> >>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same
> output.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Something is screwy with the linear system convergence, could
> be several things including either a null space problem or that your
> Jacobian that you provide is terribly wrong.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Barry
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02
> >>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true
> resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true
> resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
> >>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true
> resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Got it, fixing that, and returning to the original question,
> this is what I now get, when I use those two flags:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>  0 SNES Function norm 1.132185384796e-08
> >>>>>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true
> resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true
> resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
> >>>>>>>>>>>>>  1 SNES Function norm 2.177599365111e-12
> >>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE
> iterations 1
> >>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true
> resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true
> resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
> >>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true
> resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true
> resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true
> resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
> >>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true
> resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true
> resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true
> resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
> >>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true
> resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true
> resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true
> resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
> >>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true
> resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true
> resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true
> resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
> >>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true
> resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true
> resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true
> resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
> >>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true
> resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true
> resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true
> resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
> >>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true
> resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true
> resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
> >>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true
> resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true
> resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true
> resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
> >>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true
> resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true
> resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true
> resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
> >>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true
> resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true
> resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true
> resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
> >>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true
> resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true
> resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true
> resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
> >>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true
> resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true
> resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true
> resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
> >>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true
> resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true
> resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true
> resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
> >>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true
> resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true
> resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true
> resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
> >>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true
> resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true
> resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true
> resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
> >>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true
> resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true
> resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true
> resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
> >>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true
> resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true
> resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true
> resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
> >>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true
> resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true
> resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true
> resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
> >>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true
> resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true
> resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true
> resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
> >>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true
> resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true
> resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true
> resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
> >>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true
> resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true
> resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true
> resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
> >>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true
> resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true
> resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true
> resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
> >>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true
> resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true
> resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true
> resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
> >>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true
> resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true
> resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true
> resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
> >>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true
> resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true
> resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true
> resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
> >>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true
> resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true
> resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true
> resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
> >>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true
> resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true
> resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true
> resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
> >>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true
> resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true
> resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true
> resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
> >>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true
> resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true
> resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true
> resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
> >>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true
> resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true
> resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true
> resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
> >>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true
> resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true
> resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true
> resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
> >>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true
> resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true
> resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true
> resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
> >>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true
> resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true
> resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true
> resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
> >>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true
> resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true
> resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true
> resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
> >>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true
> resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true
> resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true
> resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
> >>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true
> resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true
> resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true
> resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
> >>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true
> resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true
> resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true
> resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
> >>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true
> resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true
> resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true
> resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
> >>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true
> resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true
> resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true
> resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
> >>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true
> resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
> >>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true
> resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
> >>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true
> resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
> >>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true
> resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
> >>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true
> resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
> >>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true
> resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
> >>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true
> resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
> >>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true
> resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
> >>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH
> iterations 47
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you
> are setting matrix entries with the first Mat argument when you should
> always set them with the second matrix argument. For example if you have
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J.
> Then at the end of the function you should call MatAssemblyBegin/End() on
> jpre then on J if J is not == jpre  see for example
> src/snes/examples/tutorials/ex1.c
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This is a minor glitch we'll get past.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Barry
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> By the way, I tried using a different petsc installation,
> and now, rather than the segmentation fault, I get the following error:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> >>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this
> object type
> >>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
> >>>>>>>>>>>>>>> [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> >>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
> >>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a
> arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
> >>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes
> --download-suitesparse=yes --download-superlu=yes
> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes
> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes
> --download-parmetis=yes --download-scalapack=yes
> >>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in
> /opt/petsc-3.5.4/src/mat/interface/matrix.c
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hmm,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since
> it is one process, type
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste
> all output and send it)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other
> options you normally use
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Barry
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Running with that flag gives me this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to
> ./blowup_batch_refine of pid 16111 on gs_air
> >>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of
> PETSc installed and the option -start_in_debugger noxterm  Once the
> debugger starts up type cont and when it crashes type where or bt  Send all
> output
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Barry
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I?m getting an error with -snes_mf_operator,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV:
> Segmentation Violation, probably memory access out of range
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on
> GNU/linux and Apple Mac OS X to find memory corruption errors
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes,
> recompile, link, and run
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a
> arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local
> --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries
> --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0
> --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate
> --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local
> --with-superlu-dir=/opt/local --with-metis-dir=/opt/local
> --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local
> --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local
> CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp
> FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp
> F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os
> FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names"
> CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os
> FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports
> --with-mpiexec=mpiexec-mpich-mp
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in
> unknown file
> >>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) -
> process 0
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct
> "enough"), hence PETSc SNES is generating a poor descent direction. You can
> try
> >>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as
> additional arguments. What happens?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Barry
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> No problem Matt, I don?t think we had previously
> discussed that output.  Here is a case where things fail.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09
> >>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12
> >>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to
> CONVERGED_SNORM_RELATIVE iterations 1
> >>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03
> >>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02
> >>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02
> >>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02
> >>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02
> >>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02
> >>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02
> >>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02
> >>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02
> >>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02
> >>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02
> >>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02
> >>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02
> >>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02
> >>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02
> >>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02
> >>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02
> >>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02
> >>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02
> >>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02
> >>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02
> >>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02
> >>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02
> >>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02
> >>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02
> >>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02
> >>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02
> >>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02
> >>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02
> >>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02
> >>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02
> >>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02
> >>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02
> >>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02
> >>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02
> >>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02
> >>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02
> >>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02
> >>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02
> >>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02
> >>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02
> >>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02
> >>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02
> >>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02
> >>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02
> >>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02
> >>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to
> DIVERGED_LINE_SEARCH iterations 45
> >>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03
> >>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03
> >>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03
> >>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03
> >>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03
> >>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03
> >>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03
> >>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03
> >>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03
> >>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03
> >>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03
> >>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03
> >>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03
> >>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03
> >>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03
> >>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03
> >>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03
> >>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03
> >>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03
> >>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03
> >>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03
> >>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03
> >>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03
> >>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03
> >>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03
> >>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03
> >>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03
> >>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03
> >>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to
> DIVERGED_LINE_SEARCH iterations 27
> >>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
> >>>>>>>>>>>>>>>>>>>>> type: newtonls
> >>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function
> evaluations=10000
> >>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50,
> solution=1e-08
> >>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
> >>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
> >>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
> >>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
> >>>>>>>>>>>>>>>>>>>>> type: bt
> >>>>>>>>>>>>>>>>>>>>> interpolation: cubic
> >>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
> >>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
> >>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08,
> absolute=1.000000e-15, lambda=1.000000e-08
> >>>>>>>>>>>>>>>>>>>>> maximum iterations=40
> >>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
> >>>>>>>>>>>>>>>>>>>>> type: gmres
> >>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified)
> Gram-Schmidt Orthogonalization with no iterative refinement
> >>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
> >>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
> >>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50,
> divergence=10000
> >>>>>>>>>>>>>>>>>>>>> left preconditioning
> >>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
> >>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
> >>>>>>>>>>>>>>>>>>>>> type: lu
> >>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
> >>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
> >>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
> >>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
> >>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
> >>>>>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
> >>>>>>>>>>>>>>>>>>>>>    type: seqaij
> >>>>>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
> >>>>>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
> >>>>>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
> >>>>>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues
> calls =0
> >>>>>>>>>>>>>>>>>>>>>      MUMPS run parameters:
> >>>>>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0
> >>>>>>>>>>>>>>>>>>>>>        PAR (host participation):            1
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):
>            1
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace
> increase): 20
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):
>               0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be
> allocated locally):0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space
> basis):          0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or
> solution):          0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):
>            -8
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential
> ordering):        1
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):
>           0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in
> inv(A)):    0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve
> phase):    0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):
>           0
> >>>>>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):
> 0.01
> >>>>>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement):
> 1.49012e-08
> >>>>>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):
>   0
> >>>>>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):
>  -1
> >>>>>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0
> >>>>>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the
> elimination after analysis):
> >>>>>>>>>>>>>>>>>>>>>          [0] 1.95838e+06
> >>>>>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the
> assembly after factorization):
> >>>>>>>>>>>>>>>>>>>>>          [0]  143924
> >>>>>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the
> elimination after factorization):
> >>>>>>>>>>>>>>>>>>>>>          [0]  1.95943e+06
> >>>>>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS
> internal data for running numerical factorization):
> >>>>>>>>>>>>>>>>>>>>>        [0] 7
> >>>>>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data
> used during numerical factorization):
> >>>>>>>>>>>>>>>>>>>>>          [0] 7
> >>>>>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this
> processor after factorization):
> >>>>>>>>>>>>>>>>>>>>>          [0] 15991
> >>>>>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the
> elimination after analysis): 1.95838e+06
> >>>>>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the
> assembly after factorization): 143924
> >>>>>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the
> elimination after factorization): 1.95943e+06
> >>>>>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34)
> (determinant): (0,0)*(2^0)
> >>>>>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for
> factors on all processors after analysis): 255801
> >>>>>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for
> factors on all processors after analysis): 127874
> >>>>>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the
> complete tree): 11
> >>>>>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete
> tree): 3996
> >>>>>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use
> after analysis): 6
> >>>>>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of
> the permuted matrix after analysis): 86
> >>>>>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to
> store the matrix factors after factorization): 255865
> >>>>>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the
> matrix factors after factorization): 127890
> >>>>>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix
> after factorization): 11
> >>>>>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19
> >>>>>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after
> factorization): 8
> >>>>>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after
> factorization): 0
> >>>>>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative
> refinement after solution): 0
> >>>>>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all
> MUMPS internal data for factorization after analysis: value on the most
> memory consuming processor): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS
> internal data for factorization after analysis: sum over all processors): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data
> allocated during factorization: value on the most memory consuming
> processor): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data
> allocated during factorization: sum over all processors): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the
> factors): 255801
> >>>>>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively
> used during factorization - value on the most memory consuming processor): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively
> used during factorization - sum over all processors): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6)
> effectively used): 0
> >>>>>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12)
> effectively used): 1
> >>>>>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of
> pivots modified by static pivoting): 0
> >>>>>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of
> null pivots encountered): 0
> >>>>>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective
> number of entries in the factors (sum over all processors)): 255865
> >>>>>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes
> of memory used during solution phase): 5, 5
> >>>>>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis
> done): 1
> >>>>>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if
> determinant is requested): 0
> >>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
> >>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
> >>>>>>>>>>>>>>>>>>>>> type: seqaij
> >>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
> >>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
> >>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues
> calls =15991
> >>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used
> is 5
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <
> knepley at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>> Barry,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid
> sequencing within my code.  I find that, in some cases, even if it can
> solve successfully on the coarsest mesh, the SNES fails, usually due to a
> line search failure, when it tries to compute along the grid sequence.
> Would you have any suggestions?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you
> give me -snes_view for the solver? I could not find it in the email thread.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line
> search, or precondition it with Richardson. It would be nice to see
> -snes_monitor
> >>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down
> the residual into fields and look at it again (if my custom residual monitor
> >>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing
> which part of the residual does not converge is key to designing the NASM
> >>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of
> this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Matt
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the
> coarse mesh, I can generally solve at all values. I do find that I need to
> do some amount of continuation to solve near the endpoint.  The problem is
> that on the coarse mesh, things are not fully resolved at all the values
> along the continuation parameter, and I would like to do refinement.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the
> intermediate continuation solutions  too.  Currently, without doing any
> grid sequence, I compute each, write it to disk, and then go on to the next
> one.  So I now need to go back an refine them.  I was thinking that perhaps
> I could refine them on the fly, dump them to disk, and use the coarse
> solution as the starting guess at the next iteration, but that would seem
> to require resetting the snes back to the coarse grid.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh
> refinement in a post processing stage, where each value of the continuation
> is parameter is loaded on the coarse mesh, and refined.  Perhaps that?s the
> most practical thing to do.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and
> create a SNES that will do the continuation
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined
> solution at this continuation point) {
> >>>>>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
> >>>>>>>>>>>>>>>>>>>>>>>    SNESSetDM()
> >>>>>>>>>>>>>>>>>>>>>>>    etc
> >>>>>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
> >>>>>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
> >>>>>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
> >>>>>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do
> with the Ufine at that continuation point
> >>>>>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
> >>>>>>>>>>>>>>>>>>>>>>> end if
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Barry
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a
> continuation problem that roughly looks like this
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>      solve with parameter p_i using solution
> from p_{i-1},
> >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver,
> for each value of parameter p_i on the coarse mesh, and then do grid
> sequencing on that.  But it appears that after doing grid sequencing on the
> initial p_0 = 0, the SNES is set to use the finer mesh.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good
> enough initial guess on the coarse level to even get convergence on the
> coarse level? First I would check if you even need the continuation (or can
> you not even solve the coarse problem without it).
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will
> need to tweak how you do the grid sequencing. I think this will work:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with
> your continuation parameter. This will all happen on the coarse mesh.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one
> solve on the coarse level and then interpolate to the next level etc.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before
> they begin their experiments is infinitely more interesting than any
> results to which their experiments lead.
> >>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> >>>>>>> -- Norbert Wiener
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> >>>> -- Norbert Wiener
> >>>
> >>
> >>
> >>
> >>
> >> --
> >> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> >> -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/efefb17f/attachment-0001.html>

From amalsahai at gmail.com  Wed Sep  9 08:24:22 2015
From: amalsahai at gmail.com (Amal Sahai)
Date: Wed, 09 Sep 2015 13:24:22 +0000
Subject: [petsc-users] Setting absolute tolerance on ksp solver
In-Reply-To: <95FB0D3D-A020-4B9A-B760-6F7A1C4AFC9A@mcs.anl.gov>
References: <CA+8mhyV4VU29WdMUSr7qXkd6gA+1h9g7u3UG4O8Jtx68pe05Xw@mail.gmail.com>
	<95FB0D3D-A020-4B9A-B760-6F7A1C4AFC9A@mcs.anl.gov>
Message-ID: <CA+8mhyUn_-hU1smpRe094SQ2eX8J-XyPuuKVBXfShqdqbN__xA@mail.gmail.com>

The problem still persists. The solver keeps iterating till the maximum
number of iterations is reached despite absolute convergence criteria being
met long before that.

Regards
Amal

On Tue, Sep 8, 2015, 22:50 Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Instead of setting rtol to default you should set it zero so it never
> gets triggered.
>
> > On Sep 8, 2015, at 10:38 PM, Amal Sahai <amalsahai at gmail.com> wrote:
> >
> > I have been setting the tolerance for a ksp solver using
> KSPSetTolerances. I would like to set my convergence criteria using only
> the absolute convergence parameter. I use the following command to set this
> up:
> >
> > KSPSetTolerances( ksp, PETSC_DEFAULT, atol, PETSC_DEFAULT, maxits)
> >
> > The problem is my ksp solver considers only the value of rtol while
> stopping the iterative solve. I tried changing the value of rtol to some
> other parameter instead of PETSC_DEFAULT (while making sure that atol is
> the more stringent condition requiring more iterations) but to no avail.
> >
> > How can I fix this?
> >
> > Regards
> > Amal
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/c193e634/attachment.html>

From knepley at gmail.com  Wed Sep  9 08:33:19 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 9 Sep 2015 08:33:19 -0500
Subject: [petsc-users] Setting absolute tolerance on ksp solver
In-Reply-To: <CA+8mhyUn_-hU1smpRe094SQ2eX8J-XyPuuKVBXfShqdqbN__xA@mail.gmail.com>
References: <CA+8mhyV4VU29WdMUSr7qXkd6gA+1h9g7u3UG4O8Jtx68pe05Xw@mail.gmail.com>
	<95FB0D3D-A020-4B9A-B760-6F7A1C4AFC9A@mcs.anl.gov>
	<CA+8mhyUn_-hU1smpRe094SQ2eX8J-XyPuuKVBXfShqdqbN__xA@mail.gmail.com>
Message-ID: <CAMYG4GmM_+W0J3sqTNqq6TgGjw5CSVobLXXgE7_3vn=euS050w@mail.gmail.com>

On Wed, Sep 9, 2015 at 8:24 AM, Amal Sahai <amalsahai at gmail.com> wrote:

> The problem still persists. The solver keeps iterating till the maximum
> number of iterations is reached despite absolute convergence criteria being
> met long before that.
>
>
Run with -ksp_view -ksp_monitor_true_residual -ksp_converged_reason

and send the output.

   Matt


> Regards
> Amal
>
> On Tue, Sep 8, 2015, 22:50 Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   Instead of setting rtol to default you should set it zero so it never
>> gets triggered.
>>
>> > On Sep 8, 2015, at 10:38 PM, Amal Sahai <amalsahai at gmail.com> wrote:
>> >
>> > I have been setting the tolerance for a ksp solver using
>> KSPSetTolerances. I would like to set my convergence criteria using only
>> the absolute convergence parameter. I use the following command to set this
>> up:
>> >
>> > KSPSetTolerances( ksp, PETSC_DEFAULT, atol, PETSC_DEFAULT, maxits)
>> >
>> > The problem is my ksp solver considers only the value of rtol while
>> stopping the iterative solve. I tried changing the value of rtol to some
>> other parameter instead of PETSC_DEFAULT (while making sure that atol is
>> the more stringent condition requiring more iterations) but to no avail.
>> >
>> > How can I fix this?
>> >
>> > Regards
>> > Amal
>>
>>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/2a186a61/attachment.html>

From jed at jedbrown.org  Wed Sep  9 09:47:12 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 09 Sep 2015 08:47:12 -0600
Subject: [petsc-users] pcfieldsplit for a composite dm with
	multiple	subfields
In-Reply-To: <DD266ECD-90C9-46A9-ADEC-40FECF3E2393@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmai l.com> <
	F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<DD266ECD-90C9-46A9-ADEC-40FECF3E2393@gmail.com>
Message-ID: <87d1xr7iq7.fsf@jedbrown.org>

Gideon Simpson <gideon.simpson at gmail.com> writes:

> I installed the gdb-apple via macports, but now it?s throwing a fit because my application has command line flags attached to it.  Is there another way to diagnose this?

GDB can take command line options via

  gdb --args ./blowup_batch_refine -your -options

unless you have the sort of heirloom version that Apple ships, in which
case you need

  gdb ./blowup_batch_refine
  [...]
  (gdb) run -your -options
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/2676b86e/attachment.pgp>

From jed at jedbrown.org  Wed Sep  9 11:05:28 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 09 Sep 2015 10:05:28 -0600
Subject: [petsc-users] Query regarding convergence of KSP solver
In-Reply-To: <CADe0GomTomy13Jm0PoVgDyoZztTSHZB+XxpG2WeoW7FkAOPAdg@mail.gmail.com>
References: <CADe0GokDkBbzV1Mp8oZXk6SYyHTnkVjPCk+F6-Ny8adJXhCcqg@mail.gmail.com>
	<87oahc7ezi.fsf@jedbrown.org>
	<CADe0GomTomy13Jm0PoVgDyoZztTSHZB+XxpG2WeoW7FkAOPAdg@mail.gmail.com>
Message-ID: <8737yn7f3r.fsf@jedbrown.org>

Please always use "reply-all" so that your messages go to the list.
This is standard mailing list etiquette.  It is important to preserve
threading for people who find this discussion later and so that we do
not waste our time re-answering the same questions that have already
been answered in private side-conversations.  You'll likely get an
answer faster that way too.

Rachit Prasad <rachitp at vt.edu> writes:

> Thank You for the response. I have already set the pre-conditioner to
> PCILU. What should I do next? Try different pre-conditioners?

Yes.  You can (a) just try stuff, (b) search with an understanding of
methods and the problem you're trying to solve, or (c) consult the
literature for your class of problem and configure the solver based on a
paper that claims to work well.

You may find these tips useful.

https://scicomp.stackexchange.com/questions/513/why-is-my-iterative-linear-solver-not-converging?lq=1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/2b1894bb/attachment.pgp>

From gideon.simpson at gmail.com  Wed Sep  9 11:14:55 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Wed, 9 Sep 2015 12:14:55 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
Message-ID: <ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>

Checking the Jacobian, this is a pretty standard output:

      Testing hand-coded Jacobian, if the ratio is O(1.e-8), the hand-coded Jacobian is probably correct.
    Finite difference Jacobian
    Hand-coded Jacobian
    Hand-coded minus finite difference Jacobian
    6.08281e-10 = ||J - Jfd||//J|| 0.000151055  = ||J - Jfd||


Things look favorable in terms of relative difference, but the absolute difference is a bit more suspect.

-gideon

> On Sep 8, 2015, at 11:49 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>> 
>> On Sep 8, 2015, at 10:28 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>> 
>> I should clarify, in that run, I ran with the analytic jacobian, but I did not use -snes_mf_operator.  If I use that flag, the performance is a bit different.  In particular, the true residual norms are not as good.  How should I interpret that?
> 
>  The "analytic Jacobians" are wrong. Of course, that is easy to say, the hard part is figuring out exactly what entries are wrong.
> 
>  You can try the following options. They are experimental so may be flaky run with the options below (but not -snes_fd or -snes_mf_operators)
> 
> -snes_check_jacobian 
> -snes_check_jacobian_view 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/9b59919c/attachment-0001.html>

From jed at jedbrown.org  Wed Sep  9 11:19:40 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 09 Sep 2015 10:19:40 -0600
Subject: [petsc-users] pcfieldsplit for a composite dm with
	multiple	subfields
In-Reply-To: <ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C7 0-FBF6-4 3BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
Message-ID: <87zj0v5zvn.fsf@jedbrown.org>

Gideon Simpson <gideon.simpson at gmail.com> writes:

> Checking the Jacobian, this is a pretty standard output:
>
>       Testing hand-coded Jacobian, if the ratio is O(1.e-8), the hand-coded Jacobian is probably correct.
>     Finite difference Jacobian
>     Hand-coded Jacobian
>     Hand-coded minus finite difference Jacobian
>     6.08281e-10 = ||J - Jfd||//J|| 0.000151055  = ||J - Jfd||
>
>
> Things look favorable in terms of relative difference, but the absolute difference is a bit more suspect.

It likely just means you have big matrix entries (perhaps due to poor nondimensionalization/scaling).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/73b67804/attachment.pgp>

From hengjiew at uci.edu  Wed Sep  9 11:59:11 2015
From: hengjiew at uci.edu (frank)
Date: Wed, 9 Sep 2015 09:59:11 -0700
Subject: [petsc-users] Parallel 3d decomposed FFT
Message-ID: <55F0655F.8010803@uci.edu>

Hi,

I am trying to use petsc's Interface to do a 3d FFT.
I created a 3d DMDA.  When I tried to call VecScatterPetsctoFFTW, I got 
error message saying "FFTW does not support parallel 3d FFT."
I understand that FFTW only supports 2d decomposition, like a 2d pencil. 
But is there any way I can use petsc to implement a 3d decomposed FFT?

Thank you,
Frank

From gideon.simpson at gmail.com  Wed Sep  9 12:08:42 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Wed, 9 Sep 2015 13:08:42 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <87zj0v5zvn.fsf@jedbrown.org>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-4 3BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<87zj0v5zvn.fsf@jedbrown.org>
Message-ID: <0E81BE5A-0A58-440B-AD8D-CBDF6E1F265F@gmail.com>

The coming back to the original problem, I?m still unsure of how to diagnose the problem.  

I am not sure if the listserv supports image attachments, but I dumped the snes_check_jacobian_view to disk and inspected the matrices with the spy command. I find that no entry in the saved matrix has absolute value larger than 10^{-2}.  There are some that have absolute value larger than 10^{-3}, but the actual values of the entries at those indices are on the order of 100 or 1000, meaning that they are small in a relative sense.

-gideon

> On Sep 9, 2015, at 12:19 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Gideon Simpson <gideon.simpson at gmail.com> writes:
> 
>> Checking the Jacobian, this is a pretty standard output:
>> 
>>      Testing hand-coded Jacobian, if the ratio is O(1.e-8), the hand-coded Jacobian is probably correct.
>>    Finite difference Jacobian
>>    Hand-coded Jacobian
>>    Hand-coded minus finite difference Jacobian
>>    6.08281e-10 = ||J - Jfd||//J|| 0.000151055  = ||J - Jfd||
>> 
>> 
>> Things look favorable in terms of relative difference, but the absolute difference is a bit more suspect.
> 
> It likely just means you have big matrix entries (perhaps due to poor nondimensionalization/scaling).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/22529cd3/attachment.html>

From bsmith at mcs.anl.gov  Wed Sep  9 12:52:51 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 9 Sep 2015 12:52:51 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
Message-ID: <BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>


  Please run with the -snes_fd and one level of grid sequencing (where Newton was having trouble converging on the second level) with -snes_monitor -ksp_monitor_true_residual and send the output. Too much speculation and not enough data.

   Barry


> On Sep 9, 2015, at 11:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> Checking the Jacobian, this is a pretty standard output:
> 
>       Testing hand-coded Jacobian, if the ratio is O(1.e-8), the hand-coded Jacobian is probably correct.
>     Finite difference Jacobian
>     Hand-coded Jacobian
>     Hand-coded minus finite difference Jacobian
>     6.08281e-10 = ||J - Jfd||//J|| 0.000151055  = ||J - Jfd||
> 
> 
> Things look favorable in terms of relative difference, but the absolute difference is a bit more suspect.
> 
> -gideon
> 
>> On Sep 8, 2015, at 11:49 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>>> 
>>> On Sep 8, 2015, at 10:28 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> I should clarify, in that run, I ran with the analytic jacobian, but I did not use -snes_mf_operator.  If I use that flag, the performance is a bit different.  In particular, the true residual norms are not as good.  How should I interpret that?
>> 
>>  The "analytic Jacobians" are wrong. Of course, that is easy to say, the hard part is figuring out exactly what entries are wrong.
>> 
>>  You can try the following options. They are experimental so may be flaky run with the options below (but not -snes_fd or -snes_mf_operators)
>> 
>> -snes_check_jacobian 
>> -snes_check_jacobian_view 
>> 
> 


From jed at jedbrown.org  Wed Sep  9 12:58:50 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 09 Sep 2015 11:58:50 -0600
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <0E81BE5A-0A58-440B-AD8D-CBDF6E1F265F@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov> <ECBF9 8CA-0C28
	-4CD9-9F52-89E167258FBE@gmail.com> <87zj0v5zvn.fsf@jedbrown.org>
	<0E81BE5A-0A58-440B-AD8D-CBDF6E1F265F@gmail.com>
Message-ID: <87r3m75vad.fsf@jedbrown.org>

Gideon Simpson <gideon.simpson at gmail.com> writes:

> The coming back to the original problem, I?m still unsure of how to diagnose the problem.  
>
> I am not sure if the listserv supports image attachments,

Yes, if small.

> but I dumped the snes_check_jacobian_view to disk and inspected the
> matrices with the spy command. I find that no entry in the saved
> matrix has absolute value larger than 10^{-2}.  There are some that
> have absolute value larger than 10^{-3}, but the actual values of the
> entries at those indices are on the order of 100 or 1000, meaning that
> they are small in a relative sense.

It's Frobenius norm and 0.000151055/6.08281e-10 = 248330.


You can try computing in quad precision, but it's possible that your
Jacobian is correct and your problem is just very nasty.  People usually
use continuations of various sorts for solving those.  For example, SNES
ex19 at high Grashof number will not converge even with grid sequencing,
but TS ex26 can solve the same problems handily.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/58b178b8/attachment.pgp>

From gideon.simpson at gmail.com  Wed Sep  9 13:48:36 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Wed, 9 Sep 2015 14:48:36 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
Message-ID: <9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>

So here?s one grid sequence with these solver flags:
-snes_max_it 100 -snes_converged_reason -snes_monitor \
-pc_type lu \
-pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1 -mat_mumps_icntl_7 6 \
-snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual

By the way, the problem, I believe is approaching something that is singular, so it is not unreasonable to think there is some kind of inherent ill conditioning in the problem.

    0 SNES Function norm 2.977789375068e-12 
      0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
    1 SNES Function norm 2.511921299174e-12 
  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
  0 SNES Function norm 5.066222213175e+03 
    0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
  1 SNES Function norm 8.484664416715e+02 
    0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
  2 SNES Function norm 6.549557200006e+02 
    0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
  3 SNES Function norm 5.770704424482e+02 
    0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
  4 SNES Function norm 5.237674957045e+02 
    0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
  5 SNES Function norm 4.753878819749e+02 
    0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
  6 SNES Function norm 4.221755368773e+02 
    0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
  7 SNES Function norm 3.806522663388e+02 
    0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
  8 SNES Function norm 3.762033603907e+02 
    0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
  9 SNES Function norm 3.758953286234e+02 
    0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
 10 SNES Function norm 3.756870388529e+02 
    0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
 11 SNES Function norm 3.728715884883e+02 
    0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
 12 SNES Function norm 3.722328158041e+02 
    0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
 13 SNES Function norm 3.720989356426e+02 
    0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
 14 SNES Function norm 3.720707403757e+02 
    0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
 15 SNES Function norm 3.720707293983e+02 
    0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
 16 SNES Function norm 3.719027845818e+02 
    0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
 17 SNES Function norm 3.714400762925e+02 
    0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
 18 SNES Function norm 3.701626928766e+02 
    0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
 19 SNES Function norm 3.556442798407e+02 
    0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
 20 SNES Function norm 3.496125058841e+02 
    0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
 21 SNES Function norm 3.146776071310e+02 
    0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
 22 SNES Function norm 2.855422195269e+02 
    0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
 23 SNES Function norm 2.806777486780e+02 
    0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
 24 SNES Function norm 2.805893466480e+02 
    0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
 25 SNES Function norm 2.805171735939e+02 
    0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
 26 SNES Function norm 2.805136061483e+02 
    0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
 27 SNES Function norm 2.805134008969e+02 
    0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
 28 SNES Function norm 2.805133530790e+02 
    0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
 29 SNES Function norm 2.805133509342e+02 
    0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
 30 SNES Function norm 2.805133508006e+02 
    0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
 31 SNES Function norm 2.805133508001e+02 
    0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
 32 SNES Function norm 2.805133507963e+02 
    0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32

-gideon

> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Please run with the -snes_fd and one level of grid sequencing (where Newton was having trouble converging on the second level) with -snes_monitor -ksp_monitor_true_residual and send the output. Too much speculation and not enough data.
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/3bebf5c4/attachment-0001.html>

From bsmith at mcs.anl.gov  Wed Sep  9 14:15:44 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 9 Sep 2015 14:15:44 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
	<9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
Message-ID: <CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>


   Thanks. I think you should try running this with __float128 (quad precision) to see what happens.

   Do you have access to a relatively recent (last couple of years) gcc (and gfortran) compiler?

   Select a new PETSC_ARCH name say arch-quad

   and run configure with the additional option -with-precision=__float128 

   not use --download-mumps or other such packages since they don't work with __float128.  If you get everything build then run
with the same options except do not use pc_factor_mat_solver_package mumps  just use the default PETSc solver LU

   Note that if your program is reading in binary files this will be slightly tricky since you will need to convert the result to __float128 format. If you use PETSc VecLoad() or PetscBinaryRead() etc you can just use the flag -binary_read_double and will read from a double binary file and automatically put the results into the PETSc __float128 matrices.

 Interpreting results. You may get convergence which might likely indicate the scaling and conditioning of your problem are very nasty but can be overcome with more precision or you may get the same lack of convergence you get now and I don't have a good explanation for that :-)

  Note: it could also be that your SNES function evaluation has a problem.



   Barry

> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> So here?s one grid sequence with these solver flags:
> -snes_max_it 100 -snes_converged_reason -snes_monitor \
> -pc_type lu \
> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1 -mat_mumps_icntl_7 6 \
> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
> 
> By the way, the problem, I believe is approaching something that is singular, so it is not unreasonable to think there is some kind of inherent ill conditioning in the problem.
> 
>     0 SNES Function norm 2.977789375068e-12 
>       0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>     1 SNES Function norm 2.511921299174e-12 
>   Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>   0 SNES Function norm 5.066222213175e+03 
>     0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
>   1 SNES Function norm 8.484664416715e+02 
>     0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
>   2 SNES Function norm 6.549557200006e+02 
>     0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
>   3 SNES Function norm 5.770704424482e+02 
>     0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
>   4 SNES Function norm 5.237674957045e+02 
>     0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
>   5 SNES Function norm 4.753878819749e+02 
>     0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
>   6 SNES Function norm 4.221755368773e+02 
>     0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
>   7 SNES Function norm 3.806522663388e+02 
>     0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
>   8 SNES Function norm 3.762033603907e+02 
>     0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
>   9 SNES Function norm 3.758953286234e+02 
>     0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
>  10 SNES Function norm 3.756870388529e+02 
>     0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
>  11 SNES Function norm 3.728715884883e+02 
>     0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
>  12 SNES Function norm 3.722328158041e+02 
>     0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
>  13 SNES Function norm 3.720989356426e+02 
>     0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
>  14 SNES Function norm 3.720707403757e+02 
>     0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
>  15 SNES Function norm 3.720707293983e+02 
>     0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
>  16 SNES Function norm 3.719027845818e+02 
>     0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
>  17 SNES Function norm 3.714400762925e+02 
>     0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
>  18 SNES Function norm 3.701626928766e+02 
>     0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
>  19 SNES Function norm 3.556442798407e+02 
>     0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
>  20 SNES Function norm 3.496125058841e+02 
>     0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
>  21 SNES Function norm 3.146776071310e+02 
>     0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
>  22 SNES Function norm 2.855422195269e+02 
>     0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
>  23 SNES Function norm 2.806777486780e+02 
>     0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
>  24 SNES Function norm 2.805893466480e+02 
>     0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
>  25 SNES Function norm 2.805171735939e+02 
>     0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
>  26 SNES Function norm 2.805136061483e+02 
>     0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
>  27 SNES Function norm 2.805134008969e+02 
>     0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
>  28 SNES Function norm 2.805133530790e+02 
>     0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
>  29 SNES Function norm 2.805133509342e+02 
>     0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
>  30 SNES Function norm 2.805133508006e+02 
>     0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
>  31 SNES Function norm 2.805133508001e+02 
>     0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
>  32 SNES Function norm 2.805133507963e+02 
>     0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
> 
> -gideon
> 
>> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  Please run with the -snes_fd and one level of grid sequencing (where Newton was having trouble converging on the second level) with -snes_monitor -ksp_monitor_true_residual and send the output. Too much speculation and not enough data.
>> 
> 


From epscodes at gmail.com  Wed Sep  9 15:57:42 2015
From: epscodes at gmail.com (Xiangdong)
Date: Wed, 9 Sep 2015 16:57:42 -0400
Subject: [petsc-users] same_nonzero_pattern in SNESSetJacobian()
Message-ID: <CAAPpcpnUTYAwG65HUXpdsYihAgyerOkfWD9m7Kh6V8Oz+QgjPA@mail.gmail.com>

Hello everyone,

As mentioned in the 3.5 changes, MatStructure argument is removed
from SNESSetJacobian(). If each time, my Jacobian always has
same_nonzero_pattern. How do I pass this information to SNES or the KSP? I
only find reuse same preconditioner, but not reuse same pattern.

Thank you.

Xiangdong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/f2022e81/attachment.html>

From knepley at gmail.com  Wed Sep  9 15:59:11 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 9 Sep 2015 15:59:11 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <65B2C6D5-47C9-49A1-9025-283168A07A0A@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<65B2C6D5-47C9-49A1-9025-283168A07A0A@gmail.com>
Message-ID: <CAMYG4G=3oEcVcssxnRhU-rEiV-sWCe32A7VTFi5Z0_MTkZwreQ@mail.gmail.com>

On Tue, Sep 8, 2015 at 10:00 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Ok, again, I?m ignorant, where do I put the VecView command to get it to
> save at some particular iterate?  Also, what do you mean by ?halves??
>

Well, you should be able to just give -snes_monitor_residual, but it does
not take arguments nicely (ugh). I can fix that, but in the meantime
you can just put VecViewFromOptions(v, NULL, "-res_vec_view") in a custom
monitor.

By halves I meant the two different problems. I want to see the residual
norms for both parts of the combined problem.

  Matt


> -gideon
>
> On Sep 8, 2015, at 10:53 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Tue, Sep 8, 2015 at 9:48 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
>> Your comment about the different pieces scaling different made me think
>> of something, and I made a slight modification.  It still fails, but the
>> conditioning looks much better, no?
>>
>
> This is great. Now its clear that the nonlinear convergence is crappy.
>
> Lets start with looking at the distribution of residuals. Take iterate 50,
> which is as converged as you get and save the residual (VecView with
> binary). Then we can load it up and play with it. I recommend
>
>   a) Norms for both halves
>
>   b) Plot of the big half
>
> What usually happens is that some small part is screwing up, but you
> have to iterate the whole system, and finding the descent direction is
> hard. If you select the problem variables, Newton will converge, and then
> the big system will converge once the problem is removed. This is the
> classic
> "coarse basis selection" problem in the nonlinear regime.
>
>    Matt
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/13a6c4ed/attachment.html>

From bsmith at mcs.anl.gov  Wed Sep  9 16:01:19 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 9 Sep 2015 16:01:19 -0500
Subject: [petsc-users] same_nonzero_pattern in SNESSetJacobian()
In-Reply-To: <CAAPpcpnUTYAwG65HUXpdsYihAgyerOkfWD9m7Kh6V8Oz+QgjPA@mail.gmail.com>
References: <CAAPpcpnUTYAwG65HUXpdsYihAgyerOkfWD9m7Kh6V8Oz+QgjPA@mail.gmail.com>
Message-ID: <0551BD7A-67D3-49FE-B3CB-DBFEEB1B48E1@mcs.anl.gov>


  Mat now keeps track of its nonzero pattern. If you do not change the pattern by inserting values then SNES/KSP will automatically keep the same pattern and optimize for it.

  Barry

> On Sep 9, 2015, at 3:57 PM, Xiangdong <epscodes at gmail.com> wrote:
> 
> Hello everyone,
> 
> As mentioned in the 3.5 changes, MatStructure argument is removed from SNESSetJacobian(). If each time, my Jacobian always has same_nonzero_pattern. How do I pass this information to SNES or the KSP? I only find reuse same preconditioner, but not reuse same pattern.
> 
> Thank you.
> 
> Xiangdong


From epscodes at gmail.com  Wed Sep  9 16:07:00 2015
From: epscodes at gmail.com (Xiangdong)
Date: Wed, 9 Sep 2015 17:07:00 -0400
Subject: [petsc-users] same_nonzero_pattern in SNESSetJacobian()
In-Reply-To: <0551BD7A-67D3-49FE-B3CB-DBFEEB1B48E1@mcs.anl.gov>
References: <CAAPpcpnUTYAwG65HUXpdsYihAgyerOkfWD9m7Kh6V8Oz+QgjPA@mail.gmail.com>
	<0551BD7A-67D3-49FE-B3CB-DBFEEB1B48E1@mcs.anl.gov>
Message-ID: <CAAPpcpnqs0227CdKSnZkGmUeoOECR2Lhur2Sh1uKWbgiXKKrWw@mail.gmail.com>

Each time I call my Jacobian function, it generates the Jacoiban matrix
from scratch by inserting entries. The result of this assembling process
gives the same nonzero pattern as in the previous iteration. Can SNES/KSP
track this too?

Thanks.

Xiangdong

On Wed, Sep 9, 2015 at 5:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Mat now keeps track of its nonzero pattern. If you do not change the
> pattern by inserting values then SNES/KSP will automatically keep the same
> pattern and optimize for it.
>
>   Barry
>
> > On Sep 9, 2015, at 3:57 PM, Xiangdong <epscodes at gmail.com> wrote:
> >
> > Hello everyone,
> >
> > As mentioned in the 3.5 changes, MatStructure argument is removed from
> SNESSetJacobian(). If each time, my Jacobian always has
> same_nonzero_pattern. How do I pass this information to SNES or the KSP? I
> only find reuse same preconditioner, but not reuse same pattern.
> >
> > Thank you.
> >
> > Xiangdong
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/55b7a339/attachment.html>

From jychang48 at gmail.com  Wed Sep  9 16:34:04 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 9 Sep 2015 15:34:04 -0600
Subject: [petsc-users] Expressing this nonlinear system in PETSc and/or
	petsc4py
Message-ID: <CAP2=TMiNLT7UK+MGGaRdmYwzvjEN4_h=hgO-5qMhjh-QOrtmYA@mail.gmail.com>

Hi everyone,

I need to solve this system of nonlinear geochemical reactions:

psi_A = c_A + c_C
psi_B = c_B + c_C
c_C = k*c_A*c_B

where psi_A and psi_B are sub components (i.e., a mixed system) of my mesh
(this was done using firedrake) and k is a constant scalar. These
quantities are known a prior, and psi_A/B was obtained from the
advection-diffusion equation (using SUPG). Given the two-field formulation
of psi_A/B, I need to obtain a three-field formulation of c_A/B/C using the
above system of equations. It should be noted that the above system of
equations are node-independent. That is, c_A/B/C of one node does not care
what psi_A/B of other nodes may be.

What?s the best strategy to go about solving this? With SciPy, i did
something like this:

#====================
from scipy.optimize import fsolve
import math

# advection-diffusion for psi_A and psi_B

# Nonlinear function for geochemical reactions
def equations(p,psi_A,psi_B,k_1):
 c_A,c_B,c_C = p
 return (c_A+c_C-psi_A,c_B+c_C-psi_B,c_C-k_1*c_A*c_B)

# Initialize
c_A_vec = np.zeros(len(psi_A.vector()))
c_B_vec = np.zeros(len(psi_A.vector()))
c_C_vec = np.zeros(len(psi_A.vector()))
k_1 = 1.0

for i in range(len(psi_A.vector())):
 c_A,c_B,c_C =
fsolve(equations,(0.0,0.0,0.0),args=(psi_A.vector()[i],psi_B.vector()[i],k_1))
 c_A_vec[i] = c_A
 c_B_vec[i] = c_B
 c_C=vec[i] = c_C

c_A = Function(Q)
c_B = Function(Q)
c_C = Function(Q)
c_A.vector()[:]=c_A_vec
c_B.vector()[:]=c_B_vec
c_C.vector()[:]=c_C_vec
#======================

The above is my temporary work-around to this issue. Basically, what I did
was I solved the equations at each node. But is there a PETSc or petsc4py
way to do this, specifically solving the above equations globally instead
of at each individual node?

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/f8094b98/attachment.html>

From bsmith at mcs.anl.gov  Wed Sep  9 16:35:52 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 9 Sep 2015 16:35:52 -0500
Subject: [petsc-users] same_nonzero_pattern in SNESSetJacobian()
In-Reply-To: <CAAPpcpnqs0227CdKSnZkGmUeoOECR2Lhur2Sh1uKWbgiXKKrWw@mail.gmail.com>
References: <CAAPpcpnUTYAwG65HUXpdsYihAgyerOkfWD9m7Kh6V8Oz+QgjPA@mail.gmail.com>
	<0551BD7A-67D3-49FE-B3CB-DBFEEB1B48E1@mcs.anl.gov>
	<CAAPpcpnqs0227CdKSnZkGmUeoOECR2Lhur2Sh1uKWbgiXKKrWw@mail.gmail.com>
Message-ID: <52C92932-7C94-4056-A46B-1CB89E01C322@mcs.anl.gov>


> On Sep 9, 2015, at 4:07 PM, Xiangdong <epscodes at gmail.com> wrote:
> 
> Each time I call my Jacobian function, it generates the Jacoiban matrix from scratch by inserting entries.

    You mean you create a new matrix? With MatCreate or MatCreateXXX()? 

    Why do this? Why not just reuse the one that is passed in? It will be more efficient.

> The result of this assembling process gives the same nonzero pattern as in the previous iteration. Can SNES/KSP track this too?

  If you are creating a new matrix, no.

  Barry

> 
> Thanks.
> 
> Xiangdong 
> 
> On Wed, Sep 9, 2015 at 5:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Mat now keeps track of its nonzero pattern. If you do not change the pattern by inserting values then SNES/KSP will automatically keep the same pattern and optimize for it.
> 
>   Barry
> 
> > On Sep 9, 2015, at 3:57 PM, Xiangdong <epscodes at gmail.com> wrote:
> >
> > Hello everyone,
> >
> > As mentioned in the 3.5 changes, MatStructure argument is removed from SNESSetJacobian(). If each time, my Jacobian always has same_nonzero_pattern. How do I pass this information to SNES or the KSP? I only find reuse same preconditioner, but not reuse same pattern.
> >
> > Thank you.
> >
> > Xiangdong
> 
> 


From gideon.simpson at gmail.com  Wed Sep  9 17:29:25 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Wed, 9 Sep 2015 18:29:25 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
	<9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
	<CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>
Message-ID: <0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>

I did a fresh install of 3.6.1 against gcc 5.2 with ?download-mpi and ?download-f2cblas with quad math support.  I then ran my code with the solver flags:
-snes_max_it 100 -snes_converged_reason -snes_monitor \
		-pc_type lu \
		-snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual -binary_read_double

I get similar results as before, but, clearly, at higher precision:

    0 SNES Function norm 2.528207888951e-12 
      0 KSP preconditioned resid norm 3.518112956043e-12 true resid norm 2.528207888951e-12 ||r(i)||/||b|| 1.000000000000e+00
      1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 1.904732438038e-41 ||r(i)||/||b|| 7.533923323167e-30
    1 SNES Function norm 2.299665925647e-23 
  Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1
  0 SNES Function norm 5.066222213176e+03 
    0 KSP preconditioned resid norm 3.135087050102e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.700962799224e-30 true resid norm 5.578720034265e-27 ||r(i)||/||b|| 1.101159759585e-30
  1 SNES Function norm 8.484697184455e+02 
    0 KSP preconditioned resid norm 2.668083368285e+02 true resid norm 8.484697184455e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 8.794084418808e-29 true resid norm 2.437641286218e-26 ||r(i)||/||b|| 2.872985603639e-29
  2 SNES Function norm 6.549559723473e+02 
    0 KSP preconditioned resid norm 1.552154272986e+02 true resid norm 6.549559723473e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.507070573051e-29 true resid norm 4.361978496405e-27 ||r(i)||/||b|| 6.659956822398e-30
  3 SNES Function norm 5.770723278332e+02 
    0 KSP preconditioned resid norm 1.348596652974e+02 true resid norm 5.770723278332e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.578833635012e-28 true resid norm 1.182025127733e-26 ||r(i)||/||b|| 2.048313652764e-29
  4 SNES Function norm 5.237702240779e+02 
    0 KSP preconditioned resid norm 1.524401223281e+02 true resid norm 5.237702240779e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.348074578423e-29 true resid norm 3.851281508331e-27 ||r(i)||/||b|| 7.352998187537e-30
  5 SNES Function norm 4.753909020039e+02 
    0 KSP preconditioned resid norm 1.864261890236e+02 true resid norm 4.753909020039e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 7.878414198958e-29 true resid norm 5.419935080688e-26 ||r(i)||/||b|| 1.140100716661e-28
  6 SNES Function norm 4.221784590918e+02 
    0 KSP preconditioned resid norm 2.550992139992e+02 true resid norm 4.221784590918e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 4.075971594113e-29 true resid norm 7.094679472303e-27 ||r(i)||/||b|| 1.680493004680e-29
  7 SNES Function norm 3.806525080588e+02 
    0 KSP preconditioned resid norm 5.183605289859e+02 true resid norm 3.806525080588e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.103573513147e-28 true resid norm 2.144465704039e-26 ||r(i)||/||b|| 5.633657098375e-29
  8 SNES Function norm 3.762054656159e+02 
    0 KSP preconditioned resid norm 5.320776223634e+03 true resid norm 3.762054656159e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.585030728289e-27 true resid norm 5.530627313578e-25 ||r(i)||/||b|| 1.470108177329e-27
  9 SNES Function norm 3.758975227011e+02 
    0 KSP preconditioned resid norm 6.252711433929e+03 true resid norm 3.758975227011e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 6.414696869631e-28 true resid norm 2.875693139366e-25 ||r(i)||/||b|| 7.650205084359e-28
 10 SNES Function norm 3.757032042685e+02 
    0 KSP preconditioned resid norm 5.890497158679e+02 true resid norm 3.757032042685e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.912199122959e-28 true resid norm 4.765625322483e-26 ||r(i)||/||b|| 1.268454798452e-28
 11 SNES Function norm 3.728798164350e+02 
    0 KSP preconditioned resid norm 1.081356529363e+03 true resid norm 3.728798164350e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.465322278036e-27 true resid norm 8.948779168104e-26 ||r(i)||/||b|| 2.399909776201e-28
 12 SNES Function norm 3.723078740346e+02 
    0 KSP preconditioned resid norm 3.745127665225e+03 true resid norm 3.723078740346e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 6.682508949012e-28 true resid norm 2.402288617952e-26 ||r(i)||/||b|| 6.452424956579e-29
 13 SNES Function norm 3.721848058457e+02 
    0 KSP preconditioned resid norm 3.495477557738e+03 true resid norm 3.721848058457e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.389031769454e-28 true resid norm 4.589073952100e-26 ||r(i)||/||b|| 1.233009483467e-28
 14 SNES Function norm 3.720227574913e+02 
    0 KSP preconditioned resid norm 1.265999878583e+04 true resid norm 3.720227574913e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.936728612764e-27 true resid norm 6.774363900864e-26 ||r(i)||/||b|| 1.820954165962e-28
 15 SNES Function norm 3.720051997704e+02 
    0 KSP preconditioned resid norm 2.912354770053e+03 true resid norm 3.720051997704e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.193330450218e-26 true resid norm 9.007685186918e-25 ||r(i)||/||b|| 2.421386903322e-27
 16 SNES Function norm 3.718945465838e+02 
    0 KSP preconditioned resid norm 1.082057529463e+03 true resid norm 3.718945465838e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.143696537658e-28 true resid norm 1.173159164030e-26 ||r(i)||/||b|| 3.154547908288e-29
 17 SNES Function norm 3.700412588801e+02 
    0 KSP preconditioned resid norm 3.676198400599e+01 true resid norm 3.700412588801e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.884191202285e-29 true resid norm 8.141362804702e-28 ||r(i)||/||b|| 2.200122988810e-30
 18 SNES Function norm 3.351964193518e+02 
    0 KSP preconditioned resid norm 1.071494896915e+02 true resid norm 3.351964193518e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.159408051956e-29 true resid norm 3.785844889574e-27 ||r(i)||/||b|| 1.129440731168e-29
 19 SNES Function norm 3.096020694819e+02 
    0 KSP preconditioned resid norm 3.361110777562e+02 true resid norm 3.096020694819e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.067163043170e-28 true resid norm 3.097548865490e-26 ||r(i)||/||b|| 1.000493591879e-28
 20 SNES Function norm 3.008463400554e+02 
    0 KSP preconditioned resid norm 7.151989169521e+01 true resid norm 3.008463400554e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.963021196462e-29 true resid norm 3.267842047327e-27 ||r(i)||/||b|| 1.086216321171e-29
 21 SNES Function norm 2.752372240183e+02 
    0 KSP preconditioned resid norm 1.737818994441e+02 true resid norm 2.752372240183e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.355051821355e-29 true resid norm 1.159582173983e-27 ||r(i)||/||b|| 4.213028154600e-30
 22 SNES Function norm 2.707638991136e+02 
    0 KSP preconditioned resid norm 4.095365990261e+02 true resid norm 2.707638991136e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.313462531085e-29 true resid norm 2.326415447553e-27 ||r(i)||/||b|| 8.592044416441e-30
 23 SNES Function norm 2.698513655244e+02 
    0 KSP preconditioned resid norm 7.571732975899e+01 true resid norm 2.698513655244e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 4.706642213758e-29 true resid norm 5.275659070101e-28 ||r(i)||/||b|| 1.955024040679e-30
 24 SNES Function norm 2.617973287453e+02 
    0 KSP preconditioned resid norm 1.781133707863e+02 true resid norm 2.617973287453e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 7.624863846438e-29 true resid norm 1.041367251217e-27 ||r(i)||/||b|| 3.977761179642e-30
 25 SNES Function norm 2.600135095660e+02 
    0 KSP preconditioned resid norm 1.377746106386e+02 true resid norm 2.600135095660e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.148291819547e-28 true resid norm 1.521069513901e-27 ||r(i)||/||b|| 5.849963397822e-30
 26 SNES Function norm 2.584547548464e+02 
    0 KSP preconditioned resid norm 1.778914250872e+02 true resid norm 2.584547548464e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.300336366195e-28 true resid norm 2.069166109298e-27 ||r(i)||/||b|| 8.005912332807e-30
 27 SNES Function norm 2.570427136855e+02 
    0 KSP preconditioned resid norm 9.377626460831e+01 true resid norm 2.570427136855e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.766637486688e-29 true resid norm 5.436682842489e-28 ||r(i)||/||b|| 2.115089264557e-30
 28 SNES Function norm 2.553675835085e+02 
    0 KSP preconditioned resid norm 3.196813637571e+02 true resid norm 2.553675835085e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.412882946976e-28 true resid norm 4.755028260087e-26 ||r(i)||/||b|| 1.862032837041e-28
 29 SNES Function norm 2.543107878046e+02 
    0 KSP preconditioned resid norm 3.373192219793e+02 true resid norm 2.543107878046e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.620724126195e-29 true resid norm 2.234654091065e-27 ||r(i)||/||b|| 8.787099085950e-30
 30 SNES Function norm 2.539436050245e+02 
    0 KSP preconditioned resid norm 3.351117240495e+02 true resid norm 2.539436050245e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.229207196291e-29 true resid norm 3.225569644851e-27 ||r(i)||/||b|| 1.270191326354e-29
 31 SNES Function norm 2.536651625665e+02 
    0 KSP preconditioned resid norm 3.687817755461e+02 true resid norm 2.536651625665e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.233376086959e-29 true resid norm 2.522505835235e-27 ||r(i)||/||b|| 9.944234398264e-30
 32 SNES Function norm 2.536237446853e+02 
    0 KSP preconditioned resid norm 6.074223711615e+02 true resid norm 2.536237446853e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.047078682230e-28 true resid norm 5.038976546196e-27 ||r(i)||/||b|| 1.986792109094e-29
 33 SNES Function norm 2.535658772687e+02 
    0 KSP preconditioned resid norm 8.774992658126e+02 true resid norm 2.535658772687e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.036350427238e-28 true resid norm 4.058678859635e-27 ||r(i)||/||b|| 1.600640789428e-29
 34 SNES Function norm 2.535415069989e+02 
    0 KSP preconditioned resid norm 1.619728779960e+03 true resid norm 2.535415069989e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.173925947581e-28 true resid norm 1.460901190689e-26 ||r(i)||/||b|| 5.761980387282e-29
 35 SNES Function norm 2.535362549500e+02 
    0 KSP preconditioned resid norm 4.120252504215e+03 true resid norm 2.535362549500e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.482004320762e-27 true resid norm 2.515988923022e-26 ||r(i)||/||b|| 9.923586366447e-29
 36 SNES Function norm 2.535350372131e+02 
    0 KSP preconditioned resid norm 7.065615798134e+03 true resid norm 2.535350372131e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.267231805260e-27 true resid norm 9.127341716497e-26 ||r(i)||/||b|| 3.600031702452e-28
 37 SNES Function norm 2.535346214155e+02 
    0 KSP preconditioned resid norm 1.200630987401e+04 true resid norm 2.535346214155e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 6.331744826877e-27 true resid norm 3.434648364475e-25 ||r(i)||/||b|| 1.354705856462e-27
 38 SNES Function norm 2.535344771151e+02 
    0 KSP preconditioned resid norm 2.045653956385e+04 true resid norm 2.535344771151e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.729474459326e-27 true resid norm 4.809771115033e-25 ||r(i)||/||b|| 1.897087595250e-27
 39 SNES Function norm 2.535344273428e+02 
    0 KSP preconditioned resid norm 3.496422852684e+04 true resid norm 2.535344273428e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.413460430546e-26 true resid norm 2.800309113539e-25 ||r(i)||/||b|| 1.104508426287e-27
 40 SNES Function norm 2.535344102918e+02 
    0 KSP preconditioned resid norm 5.986286390122e+04 true resid norm 2.535344102918e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.071099626460e-25 true resid norm 2.269549087060e-24 ||r(i)||/||b|| 8.951641256299e-27
 41 SNES Function norm 2.535344046223e+02 
    0 KSP preconditioned resid norm 1.158888344932e+05 true resid norm 2.535344046223e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 9.362165764120e-26 true resid norm 5.723929359721e-24 ||r(i)||/||b|| 2.257653894448e-26
 42 SNES Function norm 2.535344044297e+02 
    0 KSP preconditioned resid norm 6.015343562155e+05 true resid norm 2.535344044297e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.045273774078e-24 true resid norm 5.965631634103e-23 ||r(i)||/||b|| 2.352987022618e-25
 43 SNES Function norm 2.535344043720e+02 
    0 KSP preconditioned resid norm 1.032398569969e+06 true resid norm 2.535344043720e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 6.133229797301e-24 true resid norm 2.132052198244e-22 ||r(i)||/||b|| 8.409321028933e-25
 44 SNES Function norm 2.535344043525e+02 
    0 KSP preconditioned resid norm 1.771990642532e+06 true resid norm 2.535344043525e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.399736530559e-23 true resid norm 2.816123472957e-22 ||r(i)||/||b|| 1.110746085980e-24
 45 SNES Function norm 2.535344043458e+02 
    0 KSP preconditioned resid norm 3.041524303674e+06 true resid norm 2.535344043458e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 6.700523240131e-24 true resid norm 1.214940402127e-21 ||r(i)||/||b|| 4.792013948806e-24
 46 SNES Function norm 2.535344043436e+02 
    0 KSP preconditioned resid norm 5.220719804889e+06 true resid norm 2.535344043436e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 7.749472721038e-24 true resid norm 9.462448677416e-22 ||r(i)||/||b|| 3.732214845522e-24
 47 SNES Function norm 2.535344043428e+02 
    0 KSP preconditioned resid norm 8.961379294572e+06 true resid norm 2.535344043428e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.735576070962e-23 true resid norm 3.261102152175e-21 ||r(i)||/||b|| 1.286256261997e-23
 48 SNES Function norm 2.535344043425e+02 
    0 KSP preconditioned resid norm 1.538234198056e+07 true resid norm 2.535344043425e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.924165850746e-23 true resid norm 3.890747245316e-21 ||r(i)||/||b|| 1.534603264360e-23
 49 SNES Function norm 2.535344043424e+02 
    0 KSP preconditioned resid norm 2.640413180607e+07 true resid norm 2.535344043424e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.163013019454e-23 true resid norm 5.034396892612e-21 ||r(i)||/||b|| 1.985685889719e-23
Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 49
-gideon

> On Sep 9, 2015, at 3:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>   Thanks. I think you should try running this with __float128 (quad precision) to see what happens.
> 
>   Do you have access to a relatively recent (last couple of years) gcc (and gfortran) compiler?
> 
>   Select a new PETSC_ARCH name say arch-quad
> 
>   and run configure with the additional option -with-precision=__float128 
> 
>   not use --download-mumps or other such packages since they don't work with __float128.  If you get everything build then run
> with the same options except do not use pc_factor_mat_solver_package mumps  just use the default PETSc solver LU
> 
>   Note that if your program is reading in binary files this will be slightly tricky since you will need to convert the result to __float128 format. If you use PETSc VecLoad() or PetscBinaryRead() etc you can just use the flag -binary_read_double and will read from a double binary file and automatically put the results into the PETSc __float128 matrices.
> 
> Interpreting results. You may get convergence which might likely indicate the scaling and conditioning of your problem are very nasty but can be overcome with more precision or you may get the same lack of convergence you get now and I don't have a good explanation for that :-)
> 
>  Note: it could also be that your SNES function evaluation has a problem.
> 
> 
> 
>   Barry
> 
>> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> So here?s one grid sequence with these solver flags:
>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>> -pc_type lu \
>> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1 -mat_mumps_icntl_7 6 \
>> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
>> 
>> By the way, the problem, I believe is approaching something that is singular, so it is not unreasonable to think there is some kind of inherent ill conditioning in the problem.
>> 
>>    0 SNES Function norm 2.977789375068e-12 
>>      0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>>    1 SNES Function norm 2.511921299174e-12 
>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>  0 SNES Function norm 5.066222213175e+03 
>>    0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
>>  1 SNES Function norm 8.484664416715e+02 
>>    0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
>>  2 SNES Function norm 6.549557200006e+02 
>>    0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
>>  3 SNES Function norm 5.770704424482e+02 
>>    0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
>>  4 SNES Function norm 5.237674957045e+02 
>>    0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
>>  5 SNES Function norm 4.753878819749e+02 
>>    0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
>>  6 SNES Function norm 4.221755368773e+02 
>>    0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
>>  7 SNES Function norm 3.806522663388e+02 
>>    0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
>>  8 SNES Function norm 3.762033603907e+02 
>>    0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
>>  9 SNES Function norm 3.758953286234e+02 
>>    0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
>> 10 SNES Function norm 3.756870388529e+02 
>>    0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
>> 11 SNES Function norm 3.728715884883e+02 
>>    0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
>> 12 SNES Function norm 3.722328158041e+02 
>>    0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
>> 13 SNES Function norm 3.720989356426e+02 
>>    0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
>> 14 SNES Function norm 3.720707403757e+02 
>>    0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
>> 15 SNES Function norm 3.720707293983e+02 
>>    0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
>> 16 SNES Function norm 3.719027845818e+02 
>>    0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
>> 17 SNES Function norm 3.714400762925e+02 
>>    0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
>> 18 SNES Function norm 3.701626928766e+02 
>>    0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
>> 19 SNES Function norm 3.556442798407e+02 
>>    0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
>> 20 SNES Function norm 3.496125058841e+02 
>>    0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
>> 21 SNES Function norm 3.146776071310e+02 
>>    0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
>> 22 SNES Function norm 2.855422195269e+02 
>>    0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
>> 23 SNES Function norm 2.806777486780e+02 
>>    0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
>> 24 SNES Function norm 2.805893466480e+02 
>>    0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
>> 25 SNES Function norm 2.805171735939e+02 
>>    0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
>> 26 SNES Function norm 2.805136061483e+02 
>>    0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
>> 27 SNES Function norm 2.805134008969e+02 
>>    0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
>> 28 SNES Function norm 2.805133530790e+02 
>>    0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
>> 29 SNES Function norm 2.805133509342e+02 
>>    0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
>> 30 SNES Function norm 2.805133508006e+02 
>>    0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
>> 31 SNES Function norm 2.805133508001e+02 
>>    0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
>> 32 SNES Function norm 2.805133507963e+02 
>>    0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
>> 
>> -gideon
>> 
>>> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> Please run with the -snes_fd and one level of grid sequencing (where Newton was having trouble converging on the second level) with -snes_monitor -ksp_monitor_true_residual and send the output. Too much speculation and not enough data.
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/e9fe4f1b/attachment-0001.html>

From knepley at gmail.com  Wed Sep  9 17:35:42 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 9 Sep 2015 17:35:42 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
	<9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
	<CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>
	<0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>
Message-ID: <CAMYG4GmapyoFXVk9PwFn2b0W4sFPa0FcU4EwxZ2kwzTLo9_HgA@mail.gmail.com>

On Wed, Sep 9, 2015 at 5:29 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> I did a fresh install of 3.6.1 against gcc 5.2 with ?download-mpi and
> ?download-f2cblas with quad math support.  I then ran my code with the
> solver flags:
> -snes_max_it 100 -snes_converged_reason -snes_monitor \
> -pc_type lu \
> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
> -binary_read_double
>
> I get similar results as before, but, clearly, at higher precision:
>

Great! The Newton direction is just plain bad. We need to precondition it
:) So the first thing to do is find
the "problem" variables (or the possible coarse variables). We do this by
looking at the residual. This should
be fun.

   Matt


>     0 SNES Function norm 2.528207888951e-12
>       0 KSP preconditioned resid norm 3.518112956043e-12 true resid norm
> 2.528207888951e-12 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm
> 1.904732438038e-41 ||r(i)||/||b|| 7.533923323167e-30
>     1 SNES Function norm 2.299665925647e-23
>   Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1
>   0 SNES Function norm 5.066222213176e+03
>     0 KSP preconditioned resid norm 3.135087050102e+01 true resid norm
> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.700962799224e-30 true resid norm
> 5.578720034265e-27 ||r(i)||/||b|| 1.101159759585e-30
>   1 SNES Function norm 8.484697184455e+02
>     0 KSP preconditioned resid norm 2.668083368285e+02 true resid norm
> 8.484697184455e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 8.794084418808e-29 true resid norm
> 2.437641286218e-26 ||r(i)||/||b|| 2.872985603639e-29
>   2 SNES Function norm 6.549559723473e+02
>     0 KSP preconditioned resid norm 1.552154272986e+02 true resid norm
> 6.549559723473e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.507070573051e-29 true resid norm
> 4.361978496405e-27 ||r(i)||/||b|| 6.659956822398e-30
>   3 SNES Function norm 5.770723278332e+02
>     0 KSP preconditioned resid norm 1.348596652974e+02 true resid norm
> 5.770723278332e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.578833635012e-28 true resid norm
> 1.182025127733e-26 ||r(i)||/||b|| 2.048313652764e-29
>   4 SNES Function norm 5.237702240779e+02
>     0 KSP preconditioned resid norm 1.524401223281e+02 true resid norm
> 5.237702240779e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.348074578423e-29 true resid norm
> 3.851281508331e-27 ||r(i)||/||b|| 7.352998187537e-30
>   5 SNES Function norm 4.753909020039e+02
>     0 KSP preconditioned resid norm 1.864261890236e+02 true resid norm
> 4.753909020039e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 7.878414198958e-29 true resid norm
> 5.419935080688e-26 ||r(i)||/||b|| 1.140100716661e-28
>   6 SNES Function norm 4.221784590918e+02
>     0 KSP preconditioned resid norm 2.550992139992e+02 true resid norm
> 4.221784590918e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.075971594113e-29 true resid norm
> 7.094679472303e-27 ||r(i)||/||b|| 1.680493004680e-29
>   7 SNES Function norm 3.806525080588e+02
>     0 KSP preconditioned resid norm 5.183605289859e+02 true resid norm
> 3.806525080588e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.103573513147e-28 true resid norm
> 2.144465704039e-26 ||r(i)||/||b|| 5.633657098375e-29
>   8 SNES Function norm 3.762054656159e+02
>     0 KSP preconditioned resid norm 5.320776223634e+03 true resid norm
> 3.762054656159e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.585030728289e-27 true resid norm
> 5.530627313578e-25 ||r(i)||/||b|| 1.470108177329e-27
>   9 SNES Function norm 3.758975227011e+02
>     0 KSP preconditioned resid norm 6.252711433929e+03 true resid norm
> 3.758975227011e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.414696869631e-28 true resid norm
> 2.875693139366e-25 ||r(i)||/||b|| 7.650205084359e-28
>  10 SNES Function norm 3.757032042685e+02
>     0 KSP preconditioned resid norm 5.890497158679e+02 true resid norm
> 3.757032042685e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.912199122959e-28 true resid norm
> 4.765625322483e-26 ||r(i)||/||b|| 1.268454798452e-28
>  11 SNES Function norm 3.728798164350e+02
>     0 KSP preconditioned resid norm 1.081356529363e+03 true resid norm
> 3.728798164350e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.465322278036e-27 true resid norm
> 8.948779168104e-26 ||r(i)||/||b|| 2.399909776201e-28
>  12 SNES Function norm 3.723078740346e+02
>     0 KSP preconditioned resid norm 3.745127665225e+03 true resid norm
> 3.723078740346e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.682508949012e-28 true resid norm
> 2.402288617952e-26 ||r(i)||/||b|| 6.452424956579e-29
>  13 SNES Function norm 3.721848058457e+02
>     0 KSP preconditioned resid norm 3.495477557738e+03 true resid norm
> 3.721848058457e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.389031769454e-28 true resid norm
> 4.589073952100e-26 ||r(i)||/||b|| 1.233009483467e-28
>  14 SNES Function norm 3.720227574913e+02
>     0 KSP preconditioned resid norm 1.265999878583e+04 true resid norm
> 3.720227574913e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.936728612764e-27 true resid norm
> 6.774363900864e-26 ||r(i)||/||b|| 1.820954165962e-28
>  15 SNES Function norm 3.720051997704e+02
>     0 KSP preconditioned resid norm 2.912354770053e+03 true resid norm
> 3.720051997704e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.193330450218e-26 true resid norm
> 9.007685186918e-25 ||r(i)||/||b|| 2.421386903322e-27
>  16 SNES Function norm 3.718945465838e+02
>     0 KSP preconditioned resid norm 1.082057529463e+03 true resid norm
> 3.718945465838e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.143696537658e-28 true resid norm
> 1.173159164030e-26 ||r(i)||/||b|| 3.154547908288e-29
>  17 SNES Function norm 3.700412588801e+02
>     0 KSP preconditioned resid norm 3.676198400599e+01 true resid norm
> 3.700412588801e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.884191202285e-29 true resid norm
> 8.141362804702e-28 ||r(i)||/||b|| 2.200122988810e-30
>  18 SNES Function norm 3.351964193518e+02
>     0 KSP preconditioned resid norm 1.071494896915e+02 true resid norm
> 3.351964193518e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.159408051956e-29 true resid norm
> 3.785844889574e-27 ||r(i)||/||b|| 1.129440731168e-29
>  19 SNES Function norm 3.096020694819e+02
>     0 KSP preconditioned resid norm 3.361110777562e+02 true resid norm
> 3.096020694819e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.067163043170e-28 true resid norm
> 3.097548865490e-26 ||r(i)||/||b|| 1.000493591879e-28
>  20 SNES Function norm 3.008463400554e+02
>     0 KSP preconditioned resid norm 7.151989169521e+01 true resid norm
> 3.008463400554e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.963021196462e-29 true resid norm
> 3.267842047327e-27 ||r(i)||/||b|| 1.086216321171e-29
>  21 SNES Function norm 2.752372240183e+02
>     0 KSP preconditioned resid norm 1.737818994441e+02 true resid norm
> 2.752372240183e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.355051821355e-29 true resid norm
> 1.159582173983e-27 ||r(i)||/||b|| 4.213028154600e-30
>  22 SNES Function norm 2.707638991136e+02
>     0 KSP preconditioned resid norm 4.095365990261e+02 true resid norm
> 2.707638991136e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.313462531085e-29 true resid norm
> 2.326415447553e-27 ||r(i)||/||b|| 8.592044416441e-30
>  23 SNES Function norm 2.698513655244e+02
>     0 KSP preconditioned resid norm 7.571732975899e+01 true resid norm
> 2.698513655244e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.706642213758e-29 true resid norm
> 5.275659070101e-28 ||r(i)||/||b|| 1.955024040679e-30
>  24 SNES Function norm 2.617973287453e+02
>     0 KSP preconditioned resid norm 1.781133707863e+02 true resid norm
> 2.617973287453e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 7.624863846438e-29 true resid norm
> 1.041367251217e-27 ||r(i)||/||b|| 3.977761179642e-30
>  25 SNES Function norm 2.600135095660e+02
>     0 KSP preconditioned resid norm 1.377746106386e+02 true resid norm
> 2.600135095660e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.148291819547e-28 true resid norm
> 1.521069513901e-27 ||r(i)||/||b|| 5.849963397822e-30
>  26 SNES Function norm 2.584547548464e+02
>     0 KSP preconditioned resid norm 1.778914250872e+02 true resid norm
> 2.584547548464e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.300336366195e-28 true resid norm
> 2.069166109298e-27 ||r(i)||/||b|| 8.005912332807e-30
>  27 SNES Function norm 2.570427136855e+02
>     0 KSP preconditioned resid norm 9.377626460831e+01 true resid norm
> 2.570427136855e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.766637486688e-29 true resid norm
> 5.436682842489e-28 ||r(i)||/||b|| 2.115089264557e-30
>  28 SNES Function norm 2.553675835085e+02
>     0 KSP preconditioned resid norm 3.196813637571e+02 true resid norm
> 2.553675835085e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.412882946976e-28 true resid norm
> 4.755028260087e-26 ||r(i)||/||b|| 1.862032837041e-28
>  29 SNES Function norm 2.543107878046e+02
>     0 KSP preconditioned resid norm 3.373192219793e+02 true resid norm
> 2.543107878046e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.620724126195e-29 true resid norm
> 2.234654091065e-27 ||r(i)||/||b|| 8.787099085950e-30
>  30 SNES Function norm 2.539436050245e+02
>     0 KSP preconditioned resid norm 3.351117240495e+02 true resid norm
> 2.539436050245e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.229207196291e-29 true resid norm
> 3.225569644851e-27 ||r(i)||/||b|| 1.270191326354e-29
>  31 SNES Function norm 2.536651625665e+02
>     0 KSP preconditioned resid norm 3.687817755461e+02 true resid norm
> 2.536651625665e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.233376086959e-29 true resid norm
> 2.522505835235e-27 ||r(i)||/||b|| 9.944234398264e-30
>  32 SNES Function norm 2.536237446853e+02
>     0 KSP preconditioned resid norm 6.074223711615e+02 true resid norm
> 2.536237446853e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.047078682230e-28 true resid norm
> 5.038976546196e-27 ||r(i)||/||b|| 1.986792109094e-29
>  33 SNES Function norm 2.535658772687e+02
>     0 KSP preconditioned resid norm 8.774992658126e+02 true resid norm
> 2.535658772687e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.036350427238e-28 true resid norm
> 4.058678859635e-27 ||r(i)||/||b|| 1.600640789428e-29
>  34 SNES Function norm 2.535415069989e+02
>     0 KSP preconditioned resid norm 1.619728779960e+03 true resid norm
> 2.535415069989e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.173925947581e-28 true resid norm
> 1.460901190689e-26 ||r(i)||/||b|| 5.761980387282e-29
>  35 SNES Function norm 2.535362549500e+02
>     0 KSP preconditioned resid norm 4.120252504215e+03 true resid norm
> 2.535362549500e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.482004320762e-27 true resid norm
> 2.515988923022e-26 ||r(i)||/||b|| 9.923586366447e-29
>  36 SNES Function norm 2.535350372131e+02
>     0 KSP preconditioned resid norm 7.065615798134e+03 true resid norm
> 2.535350372131e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.267231805260e-27 true resid norm
> 9.127341716497e-26 ||r(i)||/||b|| 3.600031702452e-28
>  37 SNES Function norm 2.535346214155e+02
>     0 KSP preconditioned resid norm 1.200630987401e+04 true resid norm
> 2.535346214155e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.331744826877e-27 true resid norm
> 3.434648364475e-25 ||r(i)||/||b|| 1.354705856462e-27
>  38 SNES Function norm 2.535344771151e+02
>     0 KSP preconditioned resid norm 2.045653956385e+04 true resid norm
> 2.535344771151e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.729474459326e-27 true resid norm
> 4.809771115033e-25 ||r(i)||/||b|| 1.897087595250e-27
>  39 SNES Function norm 2.535344273428e+02
>     0 KSP preconditioned resid norm 3.496422852684e+04 true resid norm
> 2.535344273428e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.413460430546e-26 true resid norm
> 2.800309113539e-25 ||r(i)||/||b|| 1.104508426287e-27
>  40 SNES Function norm 2.535344102918e+02
>     0 KSP preconditioned resid norm 5.986286390122e+04 true resid norm
> 2.535344102918e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.071099626460e-25 true resid norm
> 2.269549087060e-24 ||r(i)||/||b|| 8.951641256299e-27
>  41 SNES Function norm 2.535344046223e+02
>     0 KSP preconditioned resid norm 1.158888344932e+05 true resid norm
> 2.535344046223e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 9.362165764120e-26 true resid norm
> 5.723929359721e-24 ||r(i)||/||b|| 2.257653894448e-26
>  42 SNES Function norm 2.535344044297e+02
>     0 KSP preconditioned resid norm 6.015343562155e+05 true resid norm
> 2.535344044297e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.045273774078e-24 true resid norm
> 5.965631634103e-23 ||r(i)||/||b|| 2.352987022618e-25
>  43 SNES Function norm 2.535344043720e+02
>     0 KSP preconditioned resid norm 1.032398569969e+06 true resid norm
> 2.535344043720e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.133229797301e-24 true resid norm
> 2.132052198244e-22 ||r(i)||/||b|| 8.409321028933e-25
>  44 SNES Function norm 2.535344043525e+02
>     0 KSP preconditioned resid norm 1.771990642532e+06 true resid norm
> 2.535344043525e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.399736530559e-23 true resid norm
> 2.816123472957e-22 ||r(i)||/||b|| 1.110746085980e-24
>  45 SNES Function norm 2.535344043458e+02
>     0 KSP preconditioned resid norm 3.041524303674e+06 true resid norm
> 2.535344043458e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.700523240131e-24 true resid norm
> 1.214940402127e-21 ||r(i)||/||b|| 4.792013948806e-24
>  46 SNES Function norm 2.535344043436e+02
>     0 KSP preconditioned resid norm 5.220719804889e+06 true resid norm
> 2.535344043436e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 7.749472721038e-24 true resid norm
> 9.462448677416e-22 ||r(i)||/||b|| 3.732214845522e-24
>  47 SNES Function norm 2.535344043428e+02
>     0 KSP preconditioned resid norm 8.961379294572e+06 true resid norm
> 2.535344043428e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.735576070962e-23 true resid norm
> 3.261102152175e-21 ||r(i)||/||b|| 1.286256261997e-23
>  48 SNES Function norm 2.535344043425e+02
>     0 KSP preconditioned resid norm 1.538234198056e+07 true resid norm
> 2.535344043425e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.924165850746e-23 true resid norm
> 3.890747245316e-21 ||r(i)||/||b|| 1.534603264360e-23
>  49 SNES Function norm 2.535344043424e+02
>     0 KSP preconditioned resid norm 2.640413180607e+07 true resid norm
> 2.535344043424e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.163013019454e-23 true resid norm
> 5.034396892612e-21 ||r(i)||/||b|| 1.985685889719e-23
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 49
> -gideon
>
> On Sep 9, 2015, at 3:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>   Thanks. I think you should try running this with __float128 (quad
> precision) to see what happens.
>
>   Do you have access to a relatively recent (last couple of years) gcc
> (and gfortran) compiler?
>
>   Select a new PETSC_ARCH name say arch-quad
>
>   and run configure with the additional option -with-precision=__float128
>
>   not use --download-mumps or other such packages since they don't work
> with __float128.  If you get everything build then run
> with the same options except do not use pc_factor_mat_solver_package mumps
>  just use the default PETSc solver LU
>
>   Note that if your program is reading in binary files this will be
> slightly tricky since you will need to convert the result to __float128
> format. If you use PETSc VecLoad() or PetscBinaryRead() etc you can just
> use the flag -binary_read_double and will read from a double binary file
> and automatically put the results into the PETSc __float128 matrices.
>
> Interpreting results. You may get convergence which might likely indicate
> the scaling and conditioning of your problem are very nasty but can be
> overcome with more precision or you may get the same lack of convergence
> you get now and I don't have a good explanation for that :-)
>
>  Note: it could also be that your SNES function evaluation has a problem.
>
>
>
>   Barry
>
> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> So here?s one grid sequence with these solver flags:
> -snes_max_it 100 -snes_converged_reason -snes_monitor \
> -pc_type lu \
> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1
> -mat_mumps_icntl_7 6 \
> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
>
> By the way, the problem, I believe is approaching something that is
> singular, so it is not unreasonable to think there is some kind of inherent
> ill conditioning in the problem.
>
>    0 SNES Function norm 2.977789375068e-12
>      0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm
> 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>      1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm
> 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>    1 SNES Function norm 2.511921299174e-12
>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>  0 SNES Function norm 5.066222213175e+03
>    0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm
> 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm
> 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
>  1 SNES Function norm 8.484664416715e+02
>    0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm
> 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm
> 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
>  2 SNES Function norm 6.549557200006e+02
>    0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm
> 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm
> 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
>  3 SNES Function norm 5.770704424482e+02
>    0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm
> 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm
> 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
>  4 SNES Function norm 5.237674957045e+02
>    0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm
> 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm
> 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
>  5 SNES Function norm 4.753878819749e+02
>    0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm
> 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm
> 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
>  6 SNES Function norm 4.221755368773e+02
>    0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm
> 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm
> 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
>  7 SNES Function norm 3.806522663388e+02
>    0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm
> 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm
> 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
>  8 SNES Function norm 3.762033603907e+02
>    0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm
> 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm
> 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
>  9 SNES Function norm 3.758953286234e+02
>    0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm
> 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm
> 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
> 10 SNES Function norm 3.756870388529e+02
>    0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm
> 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm
> 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
> 11 SNES Function norm 3.728715884883e+02
>    0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm
> 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm
> 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
> 12 SNES Function norm 3.722328158041e+02
>    0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm
> 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm
> 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
> 13 SNES Function norm 3.720989356426e+02
>    0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm
> 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm
> 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
> 14 SNES Function norm 3.720707403757e+02
>    0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm
> 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm
> 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
> 15 SNES Function norm 3.720707293983e+02
>    0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm
> 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm
> 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
> 16 SNES Function norm 3.719027845818e+02
>    0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm
> 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm
> 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
> 17 SNES Function norm 3.714400762925e+02
>    0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm
> 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm
> 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
> 18 SNES Function norm 3.701626928766e+02
>    0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm
> 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm
> 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
> 19 SNES Function norm 3.556442798407e+02
>    0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm
> 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm
> 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
> 20 SNES Function norm 3.496125058841e+02
>    0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm
> 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm
> 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
> 21 SNES Function norm 3.146776071310e+02
>    0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm
> 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm
> 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
> 22 SNES Function norm 2.855422195269e+02
>    0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm
> 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm
> 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
> 23 SNES Function norm 2.806777486780e+02
>    0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm
> 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm
> 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
> 24 SNES Function norm 2.805893466480e+02
>    0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm
> 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm
> 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
> 25 SNES Function norm 2.805171735939e+02
>    0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm
> 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm
> 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
> 26 SNES Function norm 2.805136061483e+02
>    0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm
> 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm
> 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
> 27 SNES Function norm 2.805134008969e+02
>    0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm
> 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm
> 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
> 28 SNES Function norm 2.805133530790e+02
>    0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm
> 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm
> 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
> 29 SNES Function norm 2.805133509342e+02
>    0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm
> 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm
> 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
> 30 SNES Function norm 2.805133508006e+02
>    0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm
> 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm
> 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
> 31 SNES Function norm 2.805133508001e+02
>    0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm
> 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm
> 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
> 32 SNES Function norm 2.805133507963e+02
>    0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm
> 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm
> 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
>
> -gideon
>
> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Please run with the -snes_fd and one level of grid sequencing (where
> Newton was having trouble converging on the second level) with
> -snes_monitor -ksp_monitor_true_residual and send the output. Too much
> speculation and not enough data.
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/657e9c61/attachment-0001.html>

From bsmith at mcs.anl.gov  Wed Sep  9 17:39:56 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 9 Sep 2015 17:39:56 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
	<9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
	<CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>
	<0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>
Message-ID: <FEA5A943-DD17-4B4C-AD1F-F1F1F04E95DD@mcs.anl.gov>


   Ok, it is able to solve the linear systems with very small residuals so I don't believe the Jacobian is single at this location.

   Please make the same run again but with the additional option -info and send all the output.

  Barry


> On Sep 9, 2015, at 5:29 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I did a fresh install of 3.6.1 against gcc 5.2 with ?download-mpi and ?download-f2cblas with quad math support.  I then ran my code with the solver flags:
> -snes_max_it 100 -snes_converged_reason -snes_monitor \
> 		-pc_type lu \
> 		-snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual -binary_read_double
> 
> I get similar results as before, but, clearly, at higher precision:
> 
>     0 SNES Function norm 2.528207888951e-12 
>       0 KSP preconditioned resid norm 3.518112956043e-12 true resid norm 2.528207888951e-12 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 1.904732438038e-41 ||r(i)||/||b|| 7.533923323167e-30
>     1 SNES Function norm 2.299665925647e-23 
>   Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1
>   0 SNES Function norm 5.066222213176e+03 
>     0 KSP preconditioned resid norm 3.135087050102e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.700962799224e-30 true resid norm 5.578720034265e-27 ||r(i)||/||b|| 1.101159759585e-30
>   1 SNES Function norm 8.484697184455e+02 
>     0 KSP preconditioned resid norm 2.668083368285e+02 true resid norm 8.484697184455e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 8.794084418808e-29 true resid norm 2.437641286218e-26 ||r(i)||/||b|| 2.872985603639e-29
>   2 SNES Function norm 6.549559723473e+02 
>     0 KSP preconditioned resid norm 1.552154272986e+02 true resid norm 6.549559723473e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.507070573051e-29 true resid norm 4.361978496405e-27 ||r(i)||/||b|| 6.659956822398e-30
>   3 SNES Function norm 5.770723278332e+02 
>     0 KSP preconditioned resid norm 1.348596652974e+02 true resid norm 5.770723278332e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.578833635012e-28 true resid norm 1.182025127733e-26 ||r(i)||/||b|| 2.048313652764e-29
>   4 SNES Function norm 5.237702240779e+02 
>     0 KSP preconditioned resid norm 1.524401223281e+02 true resid norm 5.237702240779e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.348074578423e-29 true resid norm 3.851281508331e-27 ||r(i)||/||b|| 7.352998187537e-30
>   5 SNES Function norm 4.753909020039e+02 
>     0 KSP preconditioned resid norm 1.864261890236e+02 true resid norm 4.753909020039e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 7.878414198958e-29 true resid norm 5.419935080688e-26 ||r(i)||/||b|| 1.140100716661e-28
>   6 SNES Function norm 4.221784590918e+02 
>     0 KSP preconditioned resid norm 2.550992139992e+02 true resid norm 4.221784590918e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.075971594113e-29 true resid norm 7.094679472303e-27 ||r(i)||/||b|| 1.680493004680e-29
>   7 SNES Function norm 3.806525080588e+02 
>     0 KSP preconditioned resid norm 5.183605289859e+02 true resid norm 3.806525080588e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.103573513147e-28 true resid norm 2.144465704039e-26 ||r(i)||/||b|| 5.633657098375e-29
>   8 SNES Function norm 3.762054656159e+02 
>     0 KSP preconditioned resid norm 5.320776223634e+03 true resid norm 3.762054656159e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.585030728289e-27 true resid norm 5.530627313578e-25 ||r(i)||/||b|| 1.470108177329e-27
>   9 SNES Function norm 3.758975227011e+02 
>     0 KSP preconditioned resid norm 6.252711433929e+03 true resid norm 3.758975227011e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.414696869631e-28 true resid norm 2.875693139366e-25 ||r(i)||/||b|| 7.650205084359e-28
>  10 SNES Function norm 3.757032042685e+02 
>     0 KSP preconditioned resid norm 5.890497158679e+02 true resid norm 3.757032042685e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.912199122959e-28 true resid norm 4.765625322483e-26 ||r(i)||/||b|| 1.268454798452e-28
>  11 SNES Function norm 3.728798164350e+02 
>     0 KSP preconditioned resid norm 1.081356529363e+03 true resid norm 3.728798164350e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.465322278036e-27 true resid norm 8.948779168104e-26 ||r(i)||/||b|| 2.399909776201e-28
>  12 SNES Function norm 3.723078740346e+02 
>     0 KSP preconditioned resid norm 3.745127665225e+03 true resid norm 3.723078740346e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.682508949012e-28 true resid norm 2.402288617952e-26 ||r(i)||/||b|| 6.452424956579e-29
>  13 SNES Function norm 3.721848058457e+02 
>     0 KSP preconditioned resid norm 3.495477557738e+03 true resid norm 3.721848058457e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.389031769454e-28 true resid norm 4.589073952100e-26 ||r(i)||/||b|| 1.233009483467e-28
>  14 SNES Function norm 3.720227574913e+02 
>     0 KSP preconditioned resid norm 1.265999878583e+04 true resid norm 3.720227574913e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.936728612764e-27 true resid norm 6.774363900864e-26 ||r(i)||/||b|| 1.820954165962e-28
>  15 SNES Function norm 3.720051997704e+02 
>     0 KSP preconditioned resid norm 2.912354770053e+03 true resid norm 3.720051997704e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.193330450218e-26 true resid norm 9.007685186918e-25 ||r(i)||/||b|| 2.421386903322e-27
>  16 SNES Function norm 3.718945465838e+02 
>     0 KSP preconditioned resid norm 1.082057529463e+03 true resid norm 3.718945465838e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.143696537658e-28 true resid norm 1.173159164030e-26 ||r(i)||/||b|| 3.154547908288e-29
>  17 SNES Function norm 3.700412588801e+02 
>     0 KSP preconditioned resid norm 3.676198400599e+01 true resid norm 3.700412588801e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.884191202285e-29 true resid norm 8.141362804702e-28 ||r(i)||/||b|| 2.200122988810e-30
>  18 SNES Function norm 3.351964193518e+02 
>     0 KSP preconditioned resid norm 1.071494896915e+02 true resid norm 3.351964193518e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.159408051956e-29 true resid norm 3.785844889574e-27 ||r(i)||/||b|| 1.129440731168e-29
>  19 SNES Function norm 3.096020694819e+02 
>     0 KSP preconditioned resid norm 3.361110777562e+02 true resid norm 3.096020694819e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.067163043170e-28 true resid norm 3.097548865490e-26 ||r(i)||/||b|| 1.000493591879e-28
>  20 SNES Function norm 3.008463400554e+02 
>     0 KSP preconditioned resid norm 7.151989169521e+01 true resid norm 3.008463400554e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.963021196462e-29 true resid norm 3.267842047327e-27 ||r(i)||/||b|| 1.086216321171e-29
>  21 SNES Function norm 2.752372240183e+02 
>     0 KSP preconditioned resid norm 1.737818994441e+02 true resid norm 2.752372240183e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.355051821355e-29 true resid norm 1.159582173983e-27 ||r(i)||/||b|| 4.213028154600e-30
>  22 SNES Function norm 2.707638991136e+02 
>     0 KSP preconditioned resid norm 4.095365990261e+02 true resid norm 2.707638991136e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.313462531085e-29 true resid norm 2.326415447553e-27 ||r(i)||/||b|| 8.592044416441e-30
>  23 SNES Function norm 2.698513655244e+02 
>     0 KSP preconditioned resid norm 7.571732975899e+01 true resid norm 2.698513655244e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.706642213758e-29 true resid norm 5.275659070101e-28 ||r(i)||/||b|| 1.955024040679e-30
>  24 SNES Function norm 2.617973287453e+02 
>     0 KSP preconditioned resid norm 1.781133707863e+02 true resid norm 2.617973287453e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 7.624863846438e-29 true resid norm 1.041367251217e-27 ||r(i)||/||b|| 3.977761179642e-30
>  25 SNES Function norm 2.600135095660e+02 
>     0 KSP preconditioned resid norm 1.377746106386e+02 true resid norm 2.600135095660e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.148291819547e-28 true resid norm 1.521069513901e-27 ||r(i)||/||b|| 5.849963397822e-30
>  26 SNES Function norm 2.584547548464e+02 
>     0 KSP preconditioned resid norm 1.778914250872e+02 true resid norm 2.584547548464e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.300336366195e-28 true resid norm 2.069166109298e-27 ||r(i)||/||b|| 8.005912332807e-30
>  27 SNES Function norm 2.570427136855e+02 
>     0 KSP preconditioned resid norm 9.377626460831e+01 true resid norm 2.570427136855e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.766637486688e-29 true resid norm 5.436682842489e-28 ||r(i)||/||b|| 2.115089264557e-30
>  28 SNES Function norm 2.553675835085e+02 
>     0 KSP preconditioned resid norm 3.196813637571e+02 true resid norm 2.553675835085e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.412882946976e-28 true resid norm 4.755028260087e-26 ||r(i)||/||b|| 1.862032837041e-28
>  29 SNES Function norm 2.543107878046e+02 
>     0 KSP preconditioned resid norm 3.373192219793e+02 true resid norm 2.543107878046e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.620724126195e-29 true resid norm 2.234654091065e-27 ||r(i)||/||b|| 8.787099085950e-30
>  30 SNES Function norm 2.539436050245e+02 
>     0 KSP preconditioned resid norm 3.351117240495e+02 true resid norm 2.539436050245e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.229207196291e-29 true resid norm 3.225569644851e-27 ||r(i)||/||b|| 1.270191326354e-29
>  31 SNES Function norm 2.536651625665e+02 
>     0 KSP preconditioned resid norm 3.687817755461e+02 true resid norm 2.536651625665e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.233376086959e-29 true resid norm 2.522505835235e-27 ||r(i)||/||b|| 9.944234398264e-30
>  32 SNES Function norm 2.536237446853e+02 
>     0 KSP preconditioned resid norm 6.074223711615e+02 true resid norm 2.536237446853e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.047078682230e-28 true resid norm 5.038976546196e-27 ||r(i)||/||b|| 1.986792109094e-29
>  33 SNES Function norm 2.535658772687e+02 
>     0 KSP preconditioned resid norm 8.774992658126e+02 true resid norm 2.535658772687e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.036350427238e-28 true resid norm 4.058678859635e-27 ||r(i)||/||b|| 1.600640789428e-29
>  34 SNES Function norm 2.535415069989e+02 
>     0 KSP preconditioned resid norm 1.619728779960e+03 true resid norm 2.535415069989e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.173925947581e-28 true resid norm 1.460901190689e-26 ||r(i)||/||b|| 5.761980387282e-29
>  35 SNES Function norm 2.535362549500e+02 
>     0 KSP preconditioned resid norm 4.120252504215e+03 true resid norm 2.535362549500e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.482004320762e-27 true resid norm 2.515988923022e-26 ||r(i)||/||b|| 9.923586366447e-29
>  36 SNES Function norm 2.535350372131e+02 
>     0 KSP preconditioned resid norm 7.065615798134e+03 true resid norm 2.535350372131e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.267231805260e-27 true resid norm 9.127341716497e-26 ||r(i)||/||b|| 3.600031702452e-28
>  37 SNES Function norm 2.535346214155e+02 
>     0 KSP preconditioned resid norm 1.200630987401e+04 true resid norm 2.535346214155e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.331744826877e-27 true resid norm 3.434648364475e-25 ||r(i)||/||b|| 1.354705856462e-27
>  38 SNES Function norm 2.535344771151e+02 
>     0 KSP preconditioned resid norm 2.045653956385e+04 true resid norm 2.535344771151e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.729474459326e-27 true resid norm 4.809771115033e-25 ||r(i)||/||b|| 1.897087595250e-27
>  39 SNES Function norm 2.535344273428e+02 
>     0 KSP preconditioned resid norm 3.496422852684e+04 true resid norm 2.535344273428e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.413460430546e-26 true resid norm 2.800309113539e-25 ||r(i)||/||b|| 1.104508426287e-27
>  40 SNES Function norm 2.535344102918e+02 
>     0 KSP preconditioned resid norm 5.986286390122e+04 true resid norm 2.535344102918e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.071099626460e-25 true resid norm 2.269549087060e-24 ||r(i)||/||b|| 8.951641256299e-27
>  41 SNES Function norm 2.535344046223e+02 
>     0 KSP preconditioned resid norm 1.158888344932e+05 true resid norm 2.535344046223e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 9.362165764120e-26 true resid norm 5.723929359721e-24 ||r(i)||/||b|| 2.257653894448e-26
>  42 SNES Function norm 2.535344044297e+02 
>     0 KSP preconditioned resid norm 6.015343562155e+05 true resid norm 2.535344044297e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.045273774078e-24 true resid norm 5.965631634103e-23 ||r(i)||/||b|| 2.352987022618e-25
>  43 SNES Function norm 2.535344043720e+02 
>     0 KSP preconditioned resid norm 1.032398569969e+06 true resid norm 2.535344043720e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.133229797301e-24 true resid norm 2.132052198244e-22 ||r(i)||/||b|| 8.409321028933e-25
>  44 SNES Function norm 2.535344043525e+02 
>     0 KSP preconditioned resid norm 1.771990642532e+06 true resid norm 2.535344043525e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.399736530559e-23 true resid norm 2.816123472957e-22 ||r(i)||/||b|| 1.110746085980e-24
>  45 SNES Function norm 2.535344043458e+02 
>     0 KSP preconditioned resid norm 3.041524303674e+06 true resid norm 2.535344043458e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.700523240131e-24 true resid norm 1.214940402127e-21 ||r(i)||/||b|| 4.792013948806e-24
>  46 SNES Function norm 2.535344043436e+02 
>     0 KSP preconditioned resid norm 5.220719804889e+06 true resid norm 2.535344043436e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 7.749472721038e-24 true resid norm 9.462448677416e-22 ||r(i)||/||b|| 3.732214845522e-24
>  47 SNES Function norm 2.535344043428e+02 
>     0 KSP preconditioned resid norm 8.961379294572e+06 true resid norm 2.535344043428e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.735576070962e-23 true resid norm 3.261102152175e-21 ||r(i)||/||b|| 1.286256261997e-23
>  48 SNES Function norm 2.535344043425e+02 
>     0 KSP preconditioned resid norm 1.538234198056e+07 true resid norm 2.535344043425e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.924165850746e-23 true resid norm 3.890747245316e-21 ||r(i)||/||b|| 1.534603264360e-23
>  49 SNES Function norm 2.535344043424e+02 
>     0 KSP preconditioned resid norm 2.640413180607e+07 true resid norm 2.535344043424e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.163013019454e-23 true resid norm 5.034396892612e-21 ||r(i)||/||b|| 1.985685889719e-23
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 49
> -gideon
> 
>> On Sep 9, 2015, at 3:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>   Thanks. I think you should try running this with __float128 (quad precision) to see what happens.
>> 
>>   Do you have access to a relatively recent (last couple of years) gcc (and gfortran) compiler?
>> 
>>   Select a new PETSC_ARCH name say arch-quad
>> 
>>   and run configure with the additional option -with-precision=__float128 
>> 
>>   not use --download-mumps or other such packages since they don't work with __float128.  If you get everything build then run
>> with the same options except do not use pc_factor_mat_solver_package mumps  just use the default PETSc solver LU
>> 
>>   Note that if your program is reading in binary files this will be slightly tricky since you will need to convert the result to __float128 format. If you use PETSc VecLoad() or PetscBinaryRead() etc you can just use the flag -binary_read_double and will read from a double binary file and automatically put the results into the PETSc __float128 matrices.
>> 
>> Interpreting results. You may get convergence which might likely indicate the scaling and conditioning of your problem are very nasty but can be overcome with more precision or you may get the same lack of convergence you get now and I don't have a good explanation for that :-)
>> 
>>  Note: it could also be that your SNES function evaluation has a problem.
>> 
>> 
>> 
>>   Barry
>> 
>>> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> So here?s one grid sequence with these solver flags:
>>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>>> -pc_type lu \
>>> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1 -mat_mumps_icntl_7 6 \
>>> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
>>> 
>>> By the way, the problem, I believe is approaching something that is singular, so it is not unreasonable to think there is some kind of inherent ill conditioning in the problem.
>>> 
>>>    0 SNES Function norm 2.977789375068e-12 
>>>      0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>>>    1 SNES Function norm 2.511921299174e-12 
>>>  Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>  0 SNES Function norm 5.066222213175e+03 
>>>    0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
>>>  1 SNES Function norm 8.484664416715e+02 
>>>    0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
>>>  2 SNES Function norm 6.549557200006e+02 
>>>    0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
>>>  3 SNES Function norm 5.770704424482e+02 
>>>    0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
>>>  4 SNES Function norm 5.237674957045e+02 
>>>    0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
>>>  5 SNES Function norm 4.753878819749e+02 
>>>    0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
>>>  6 SNES Function norm 4.221755368773e+02 
>>>    0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
>>>  7 SNES Function norm 3.806522663388e+02 
>>>    0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
>>>  8 SNES Function norm 3.762033603907e+02 
>>>    0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
>>>  9 SNES Function norm 3.758953286234e+02 
>>>    0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
>>> 10 SNES Function norm 3.756870388529e+02 
>>>    0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
>>> 11 SNES Function norm 3.728715884883e+02 
>>>    0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
>>> 12 SNES Function norm 3.722328158041e+02 
>>>    0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
>>> 13 SNES Function norm 3.720989356426e+02 
>>>    0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
>>> 14 SNES Function norm 3.720707403757e+02 
>>>    0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
>>> 15 SNES Function norm 3.720707293983e+02 
>>>    0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
>>> 16 SNES Function norm 3.719027845818e+02 
>>>    0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
>>> 17 SNES Function norm 3.714400762925e+02 
>>>    0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
>>> 18 SNES Function norm 3.701626928766e+02 
>>>    0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
>>> 19 SNES Function norm 3.556442798407e+02 
>>>    0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
>>> 20 SNES Function norm 3.496125058841e+02 
>>>    0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
>>> 21 SNES Function norm 3.146776071310e+02 
>>>    0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
>>> 22 SNES Function norm 2.855422195269e+02 
>>>    0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
>>> 23 SNES Function norm 2.806777486780e+02 
>>>    0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
>>> 24 SNES Function norm 2.805893466480e+02 
>>>    0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
>>> 25 SNES Function norm 2.805171735939e+02 
>>>    0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
>>> 26 SNES Function norm 2.805136061483e+02 
>>>    0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
>>> 27 SNES Function norm 2.805134008969e+02 
>>>    0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
>>> 28 SNES Function norm 2.805133530790e+02 
>>>    0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
>>> 29 SNES Function norm 2.805133509342e+02 
>>>    0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
>>> 30 SNES Function norm 2.805133508006e+02 
>>>    0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
>>> 31 SNES Function norm 2.805133508001e+02 
>>>    0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
>>> 32 SNES Function norm 2.805133507963e+02 
>>>    0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
>>> 
>>> -gideon
>>> 
>>>> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> Please run with the -snes_fd and one level of grid sequencing (where Newton was having trouble converging on the second level) with -snes_monitor -ksp_monitor_true_residual and send the output. Too much speculation and not enough data.
>>>> 
>>> 
>> 
> 


From knepley at gmail.com  Wed Sep  9 17:41:16 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 9 Sep 2015 17:41:16 -0500
Subject: [petsc-users] Expressing this nonlinear system in PETSc and/or
	petsc4py
In-Reply-To: <CAP2=TMiNLT7UK+MGGaRdmYwzvjEN4_h=hgO-5qMhjh-QOrtmYA@mail.gmail.com>
References: <CAP2=TMiNLT7UK+MGGaRdmYwzvjEN4_h=hgO-5qMhjh-QOrtmYA@mail.gmail.com>
Message-ID: <CAMYG4GkhhVyUSWzRnwWYAG93UuWPUR-q_1-=QsZXe15pS3oV+Q@mail.gmail.com>

On Wed, Sep 9, 2015 at 4:34 PM, Justin Chang <jychang48 at gmail.com> wrote:

> Hi everyone,
>
> I need to solve this system of nonlinear geochemical reactions:
>
> psi_A = c_A + c_C
> psi_B = c_B + c_C
> c_C = k*c_A*c_B
>
> where psi_A and psi_B are sub components (i.e., a mixed system) of my mesh
> (this was done using firedrake) and k is a constant scalar. These
> quantities are known a prior, and psi_A/B was obtained from the
> advection-diffusion equation (using SUPG). Given the two-field formulation
> of psi_A/B, I need to obtain a three-field formulation of c_A/B/C using the
> above system of equations. It should be noted that the above system of
> equations are node-independent. That is, c_A/B/C of one node does not care
> what psi_A/B of other nodes may be.
>
> What?s the best strategy to go about solving this? With SciPy, i did
> something like this:
>
> #====================
> from scipy.optimize import fsolve
> import math
>
> # advection-diffusion for psi_A and psi_B
>
> # Nonlinear function for geochemical reactions
> def equations(p,psi_A,psi_B,k_1):
>  c_A,c_B,c_C = p
>  return (c_A+c_C-psi_A,c_B+c_C-psi_B,c_C-k_1*c_A*c_B)
>
> # Initialize
> c_A_vec = np.zeros(len(psi_A.vector()))
> c_B_vec = np.zeros(len(psi_A.vector()))
> c_C_vec = np.zeros(len(psi_A.vector()))
> k_1 = 1.0
>
> for i in range(len(psi_A.vector())):
>  c_A,c_B,c_C =
> fsolve(equations,(0.0,0.0,0.0),args=(psi_A.vector()[i],psi_B.vector()[i],k_1))
>  c_A_vec[i] = c_A
>  c_B_vec[i] = c_B
>  c_C=vec[i] = c_C
>
> c_A = Function(Q)
> c_B = Function(Q)
> c_C = Function(Q)
> c_A.vector()[:]=c_A_vec
> c_B.vector()[:]=c_B_vec
> c_C.vector()[:]=c_C_vec
> #======================
>
> The above is my temporary work-around to this issue. Basically, what I did
> was I solved the equations at each node. But is there a PETSc or petsc4py
> way to do this, specifically solving the above equations globally instead
> of at each individual node?
>

1) I had the same thing in DFT where I had to solve a local equation for
the screening length at each node

2) I would use a SNES

3) If you think there is variability in the solve, then do each
individually. Otherwise,
    you can easily do a Newton with a diagonal Jacobian.

4) I would eliminate c_C since its so easy

5) These look quadratic. Can't you solve this analytically?

  Thanks,

     Matt


>
> Thanks,
> Justin
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/047466bc/attachment-0001.html>

From gideon.simpson at gmail.com  Wed Sep  9 22:08:36 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Wed, 9 Sep 2015 23:08:36 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <FEA5A943-DD17-4B4C-AD1F-F1F1F04E95DD@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
	<9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
	<CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>
	<0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>
	<FEA5A943-DD17-4B4C-AD1F-F1F1F04E95DD@mcs.anl.gov>
Message-ID: <E6733B83-B018-4B4C-BC93-41AE8844D6DB@gmail.com>

The output is 55 megs.  Is there some specific portion you?re looking for? 

-gideon

> On Sep 9, 2015, at 6:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>   Ok, it is able to solve the linear systems with very small residuals so I don't believe the Jacobian is single at this location.
> 
>   Please make the same run again but with the additional option -info and send all the output.
> 
>  Barry
> 
> 
>> On Sep 9, 2015, at 5:29 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> I did a fresh install of 3.6.1 against gcc 5.2 with ?download-mpi and ?download-f2cblas with quad math support.  I then ran my code with the solver flags:
>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>> 		-pc_type lu \
>> 		-snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual -binary_read_double
>> 
>> I get similar results as before, but, clearly, at higher precision:
>> 
>>    0 SNES Function norm 2.528207888951e-12 
>>      0 KSP preconditioned resid norm 3.518112956043e-12 true resid norm 2.528207888951e-12 ||r(i)||/||b|| 1.000000000000e+00
>>      1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 1.904732438038e-41 ||r(i)||/||b|| 7.533923323167e-30
>>    1 SNES Function norm 2.299665925647e-23 
>>  Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1
>>  0 SNES Function norm 5.066222213176e+03 
>>    0 KSP preconditioned resid norm 3.135087050102e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.700962799224e-30 true resid norm 5.578720034265e-27 ||r(i)||/||b|| 1.101159759585e-30
>>  1 SNES Function norm 8.484697184455e+02 
>>    0 KSP preconditioned resid norm 2.668083368285e+02 true resid norm 8.484697184455e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 8.794084418808e-29 true resid norm 2.437641286218e-26 ||r(i)||/||b|| 2.872985603639e-29
>>  2 SNES Function norm 6.549559723473e+02 
>>    0 KSP preconditioned resid norm 1.552154272986e+02 true resid norm 6.549559723473e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.507070573051e-29 true resid norm 4.361978496405e-27 ||r(i)||/||b|| 6.659956822398e-30
>>  3 SNES Function norm 5.770723278332e+02 
>>    0 KSP preconditioned resid norm 1.348596652974e+02 true resid norm 5.770723278332e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.578833635012e-28 true resid norm 1.182025127733e-26 ||r(i)||/||b|| 2.048313652764e-29
>>  4 SNES Function norm 5.237702240779e+02 
>>    0 KSP preconditioned resid norm 1.524401223281e+02 true resid norm 5.237702240779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.348074578423e-29 true resid norm 3.851281508331e-27 ||r(i)||/||b|| 7.352998187537e-30
>>  5 SNES Function norm 4.753909020039e+02 
>>    0 KSP preconditioned resid norm 1.864261890236e+02 true resid norm 4.753909020039e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 7.878414198958e-29 true resid norm 5.419935080688e-26 ||r(i)||/||b|| 1.140100716661e-28
>>  6 SNES Function norm 4.221784590918e+02 
>>    0 KSP preconditioned resid norm 2.550992139992e+02 true resid norm 4.221784590918e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 4.075971594113e-29 true resid norm 7.094679472303e-27 ||r(i)||/||b|| 1.680493004680e-29
>>  7 SNES Function norm 3.806525080588e+02 
>>    0 KSP preconditioned resid norm 5.183605289859e+02 true resid norm 3.806525080588e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.103573513147e-28 true resid norm 2.144465704039e-26 ||r(i)||/||b|| 5.633657098375e-29
>>  8 SNES Function norm 3.762054656159e+02 
>>    0 KSP preconditioned resid norm 5.320776223634e+03 true resid norm 3.762054656159e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.585030728289e-27 true resid norm 5.530627313578e-25 ||r(i)||/||b|| 1.470108177329e-27
>>  9 SNES Function norm 3.758975227011e+02 
>>    0 KSP preconditioned resid norm 6.252711433929e+03 true resid norm 3.758975227011e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 6.414696869631e-28 true resid norm 2.875693139366e-25 ||r(i)||/||b|| 7.650205084359e-28
>> 10 SNES Function norm 3.757032042685e+02 
>>    0 KSP preconditioned resid norm 5.890497158679e+02 true resid norm 3.757032042685e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.912199122959e-28 true resid norm 4.765625322483e-26 ||r(i)||/||b|| 1.268454798452e-28
>> 11 SNES Function norm 3.728798164350e+02 
>>    0 KSP preconditioned resid norm 1.081356529363e+03 true resid norm 3.728798164350e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.465322278036e-27 true resid norm 8.948779168104e-26 ||r(i)||/||b|| 2.399909776201e-28
>> 12 SNES Function norm 3.723078740346e+02 
>>    0 KSP preconditioned resid norm 3.745127665225e+03 true resid norm 3.723078740346e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 6.682508949012e-28 true resid norm 2.402288617952e-26 ||r(i)||/||b|| 6.452424956579e-29
>> 13 SNES Function norm 3.721848058457e+02 
>>    0 KSP preconditioned resid norm 3.495477557738e+03 true resid norm 3.721848058457e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.389031769454e-28 true resid norm 4.589073952100e-26 ||r(i)||/||b|| 1.233009483467e-28
>> 14 SNES Function norm 3.720227574913e+02 
>>    0 KSP preconditioned resid norm 1.265999878583e+04 true resid norm 3.720227574913e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.936728612764e-27 true resid norm 6.774363900864e-26 ||r(i)||/||b|| 1.820954165962e-28
>> 15 SNES Function norm 3.720051997704e+02 
>>    0 KSP preconditioned resid norm 2.912354770053e+03 true resid norm 3.720051997704e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.193330450218e-26 true resid norm 9.007685186918e-25 ||r(i)||/||b|| 2.421386903322e-27
>> 16 SNES Function norm 3.718945465838e+02 
>>    0 KSP preconditioned resid norm 1.082057529463e+03 true resid norm 3.718945465838e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.143696537658e-28 true resid norm 1.173159164030e-26 ||r(i)||/||b|| 3.154547908288e-29
>> 17 SNES Function norm 3.700412588801e+02 
>>    0 KSP preconditioned resid norm 3.676198400599e+01 true resid norm 3.700412588801e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.884191202285e-29 true resid norm 8.141362804702e-28 ||r(i)||/||b|| 2.200122988810e-30
>> 18 SNES Function norm 3.351964193518e+02 
>>    0 KSP preconditioned resid norm 1.071494896915e+02 true resid norm 3.351964193518e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 5.159408051956e-29 true resid norm 3.785844889574e-27 ||r(i)||/||b|| 1.129440731168e-29
>> 19 SNES Function norm 3.096020694819e+02 
>>    0 KSP preconditioned resid norm 3.361110777562e+02 true resid norm 3.096020694819e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.067163043170e-28 true resid norm 3.097548865490e-26 ||r(i)||/||b|| 1.000493591879e-28
>> 20 SNES Function norm 3.008463400554e+02 
>>    0 KSP preconditioned resid norm 7.151989169521e+01 true resid norm 3.008463400554e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.963021196462e-29 true resid norm 3.267842047327e-27 ||r(i)||/||b|| 1.086216321171e-29
>> 21 SNES Function norm 2.752372240183e+02 
>>    0 KSP preconditioned resid norm 1.737818994441e+02 true resid norm 2.752372240183e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.355051821355e-29 true resid norm 1.159582173983e-27 ||r(i)||/||b|| 4.213028154600e-30
>> 22 SNES Function norm 2.707638991136e+02 
>>    0 KSP preconditioned resid norm 4.095365990261e+02 true resid norm 2.707638991136e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.313462531085e-29 true resid norm 2.326415447553e-27 ||r(i)||/||b|| 8.592044416441e-30
>> 23 SNES Function norm 2.698513655244e+02 
>>    0 KSP preconditioned resid norm 7.571732975899e+01 true resid norm 2.698513655244e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 4.706642213758e-29 true resid norm 5.275659070101e-28 ||r(i)||/||b|| 1.955024040679e-30
>> 24 SNES Function norm 2.617973287453e+02 
>>    0 KSP preconditioned resid norm 1.781133707863e+02 true resid norm 2.617973287453e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 7.624863846438e-29 true resid norm 1.041367251217e-27 ||r(i)||/||b|| 3.977761179642e-30
>> 25 SNES Function norm 2.600135095660e+02 
>>    0 KSP preconditioned resid norm 1.377746106386e+02 true resid norm 2.600135095660e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.148291819547e-28 true resid norm 1.521069513901e-27 ||r(i)||/||b|| 5.849963397822e-30
>> 26 SNES Function norm 2.584547548464e+02 
>>    0 KSP preconditioned resid norm 1.778914250872e+02 true resid norm 2.584547548464e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.300336366195e-28 true resid norm 2.069166109298e-27 ||r(i)||/||b|| 8.005912332807e-30
>> 27 SNES Function norm 2.570427136855e+02 
>>    0 KSP preconditioned resid norm 9.377626460831e+01 true resid norm 2.570427136855e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 5.766637486688e-29 true resid norm 5.436682842489e-28 ||r(i)||/||b|| 2.115089264557e-30
>> 28 SNES Function norm 2.553675835085e+02 
>>    0 KSP preconditioned resid norm 3.196813637571e+02 true resid norm 2.553675835085e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 5.412882946976e-28 true resid norm 4.755028260087e-26 ||r(i)||/||b|| 1.862032837041e-28
>> 29 SNES Function norm 2.543107878046e+02 
>>    0 KSP preconditioned resid norm 3.373192219793e+02 true resid norm 2.543107878046e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.620724126195e-29 true resid norm 2.234654091065e-27 ||r(i)||/||b|| 8.787099085950e-30
>> 30 SNES Function norm 2.539436050245e+02 
>>    0 KSP preconditioned resid norm 3.351117240495e+02 true resid norm 2.539436050245e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.229207196291e-29 true resid norm 3.225569644851e-27 ||r(i)||/||b|| 1.270191326354e-29
>> 31 SNES Function norm 2.536651625665e+02 
>>    0 KSP preconditioned resid norm 3.687817755461e+02 true resid norm 2.536651625665e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.233376086959e-29 true resid norm 2.522505835235e-27 ||r(i)||/||b|| 9.944234398264e-30
>> 32 SNES Function norm 2.536237446853e+02 
>>    0 KSP preconditioned resid norm 6.074223711615e+02 true resid norm 2.536237446853e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.047078682230e-28 true resid norm 5.038976546196e-27 ||r(i)||/||b|| 1.986792109094e-29
>> 33 SNES Function norm 2.535658772687e+02 
>>    0 KSP preconditioned resid norm 8.774992658126e+02 true resid norm 2.535658772687e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.036350427238e-28 true resid norm 4.058678859635e-27 ||r(i)||/||b|| 1.600640789428e-29
>> 34 SNES Function norm 2.535415069989e+02 
>>    0 KSP preconditioned resid norm 1.619728779960e+03 true resid norm 2.535415069989e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.173925947581e-28 true resid norm 1.460901190689e-26 ||r(i)||/||b|| 5.761980387282e-29
>> 35 SNES Function norm 2.535362549500e+02 
>>    0 KSP preconditioned resid norm 4.120252504215e+03 true resid norm 2.535362549500e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.482004320762e-27 true resid norm 2.515988923022e-26 ||r(i)||/||b|| 9.923586366447e-29
>> 36 SNES Function norm 2.535350372131e+02 
>>    0 KSP preconditioned resid norm 7.065615798134e+03 true resid norm 2.535350372131e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.267231805260e-27 true resid norm 9.127341716497e-26 ||r(i)||/||b|| 3.600031702452e-28
>> 37 SNES Function norm 2.535346214155e+02 
>>    0 KSP preconditioned resid norm 1.200630987401e+04 true resid norm 2.535346214155e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 6.331744826877e-27 true resid norm 3.434648364475e-25 ||r(i)||/||b|| 1.354705856462e-27
>> 38 SNES Function norm 2.535344771151e+02 
>>    0 KSP preconditioned resid norm 2.045653956385e+04 true resid norm 2.535344771151e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.729474459326e-27 true resid norm 4.809771115033e-25 ||r(i)||/||b|| 1.897087595250e-27
>> 39 SNES Function norm 2.535344273428e+02 
>>    0 KSP preconditioned resid norm 3.496422852684e+04 true resid norm 2.535344273428e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.413460430546e-26 true resid norm 2.800309113539e-25 ||r(i)||/||b|| 1.104508426287e-27
>> 40 SNES Function norm 2.535344102918e+02 
>>    0 KSP preconditioned resid norm 5.986286390122e+04 true resid norm 2.535344102918e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.071099626460e-25 true resid norm 2.269549087060e-24 ||r(i)||/||b|| 8.951641256299e-27
>> 41 SNES Function norm 2.535344046223e+02 
>>    0 KSP preconditioned resid norm 1.158888344932e+05 true resid norm 2.535344046223e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 9.362165764120e-26 true resid norm 5.723929359721e-24 ||r(i)||/||b|| 2.257653894448e-26
>> 42 SNES Function norm 2.535344044297e+02 
>>    0 KSP preconditioned resid norm 6.015343562155e+05 true resid norm 2.535344044297e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.045273774078e-24 true resid norm 5.965631634103e-23 ||r(i)||/||b|| 2.352987022618e-25
>> 43 SNES Function norm 2.535344043720e+02 
>>    0 KSP preconditioned resid norm 1.032398569969e+06 true resid norm 2.535344043720e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 6.133229797301e-24 true resid norm 2.132052198244e-22 ||r(i)||/||b|| 8.409321028933e-25
>> 44 SNES Function norm 2.535344043525e+02 
>>    0 KSP preconditioned resid norm 1.771990642532e+06 true resid norm 2.535344043525e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 1.399736530559e-23 true resid norm 2.816123472957e-22 ||r(i)||/||b|| 1.110746085980e-24
>> 45 SNES Function norm 2.535344043458e+02 
>>    0 KSP preconditioned resid norm 3.041524303674e+06 true resid norm 2.535344043458e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 6.700523240131e-24 true resid norm 1.214940402127e-21 ||r(i)||/||b|| 4.792013948806e-24
>> 46 SNES Function norm 2.535344043436e+02 
>>    0 KSP preconditioned resid norm 5.220719804889e+06 true resid norm 2.535344043436e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 7.749472721038e-24 true resid norm 9.462448677416e-22 ||r(i)||/||b|| 3.732214845522e-24
>> 47 SNES Function norm 2.535344043428e+02 
>>    0 KSP preconditioned resid norm 8.961379294572e+06 true resid norm 2.535344043428e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.735576070962e-23 true resid norm 3.261102152175e-21 ||r(i)||/||b|| 1.286256261997e-23
>> 48 SNES Function norm 2.535344043425e+02 
>>    0 KSP preconditioned resid norm 1.538234198056e+07 true resid norm 2.535344043425e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 5.924165850746e-23 true resid norm 3.890747245316e-21 ||r(i)||/||b|| 1.534603264360e-23
>> 49 SNES Function norm 2.535344043424e+02 
>>    0 KSP preconditioned resid norm 2.640413180607e+07 true resid norm 2.535344043424e+02 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 2.163013019454e-23 true resid norm 5.034396892612e-21 ||r(i)||/||b|| 1.985685889719e-23
>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 49
>> -gideon
>> 
>>> On Sep 9, 2015, at 3:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>>  Thanks. I think you should try running this with __float128 (quad precision) to see what happens.
>>> 
>>>  Do you have access to a relatively recent (last couple of years) gcc (and gfortran) compiler?
>>> 
>>>  Select a new PETSC_ARCH name say arch-quad
>>> 
>>>  and run configure with the additional option -with-precision=__float128 
>>> 
>>>  not use --download-mumps or other such packages since they don't work with __float128.  If you get everything build then run
>>> with the same options except do not use pc_factor_mat_solver_package mumps  just use the default PETSc solver LU
>>> 
>>>  Note that if your program is reading in binary files this will be slightly tricky since you will need to convert the result to __float128 format. If you use PETSc VecLoad() or PetscBinaryRead() etc you can just use the flag -binary_read_double and will read from a double binary file and automatically put the results into the PETSc __float128 matrices.
>>> 
>>> Interpreting results. You may get convergence which might likely indicate the scaling and conditioning of your problem are very nasty but can be overcome with more precision or you may get the same lack of convergence you get now and I don't have a good explanation for that :-)
>>> 
>>> Note: it could also be that your SNES function evaluation has a problem.
>>> 
>>> 
>>> 
>>>  Barry
>>> 
>>>> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> So here?s one grid sequence with these solver flags:
>>>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>>>> -pc_type lu \
>>>> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1 -mat_mumps_icntl_7 6 \
>>>> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
>>>> 
>>>> By the way, the problem, I believe is approaching something that is singular, so it is not unreasonable to think there is some kind of inherent ill conditioning in the problem.
>>>> 
>>>>   0 SNES Function norm 2.977789375068e-12 
>>>>     0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>>>>   1 SNES Function norm 2.511921299174e-12 
>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>> 0 SNES Function norm 5.066222213175e+03 
>>>>   0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
>>>> 1 SNES Function norm 8.484664416715e+02 
>>>>   0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
>>>> 2 SNES Function norm 6.549557200006e+02 
>>>>   0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
>>>> 3 SNES Function norm 5.770704424482e+02 
>>>>   0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
>>>> 4 SNES Function norm 5.237674957045e+02 
>>>>   0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
>>>> 5 SNES Function norm 4.753878819749e+02 
>>>>   0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
>>>> 6 SNES Function norm 4.221755368773e+02 
>>>>   0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
>>>> 7 SNES Function norm 3.806522663388e+02 
>>>>   0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
>>>> 8 SNES Function norm 3.762033603907e+02 
>>>>   0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
>>>> 9 SNES Function norm 3.758953286234e+02 
>>>>   0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
>>>> 10 SNES Function norm 3.756870388529e+02 
>>>>   0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
>>>> 11 SNES Function norm 3.728715884883e+02 
>>>>   0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
>>>> 12 SNES Function norm 3.722328158041e+02 
>>>>   0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
>>>> 13 SNES Function norm 3.720989356426e+02 
>>>>   0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
>>>> 14 SNES Function norm 3.720707403757e+02 
>>>>   0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
>>>> 15 SNES Function norm 3.720707293983e+02 
>>>>   0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
>>>> 16 SNES Function norm 3.719027845818e+02 
>>>>   0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
>>>> 17 SNES Function norm 3.714400762925e+02 
>>>>   0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
>>>> 18 SNES Function norm 3.701626928766e+02 
>>>>   0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
>>>> 19 SNES Function norm 3.556442798407e+02 
>>>>   0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
>>>> 20 SNES Function norm 3.496125058841e+02 
>>>>   0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
>>>> 21 SNES Function norm 3.146776071310e+02 
>>>>   0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
>>>> 22 SNES Function norm 2.855422195269e+02 
>>>>   0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
>>>> 23 SNES Function norm 2.806777486780e+02 
>>>>   0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
>>>> 24 SNES Function norm 2.805893466480e+02 
>>>>   0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
>>>> 25 SNES Function norm 2.805171735939e+02 
>>>>   0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
>>>> 26 SNES Function norm 2.805136061483e+02 
>>>>   0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
>>>> 27 SNES Function norm 2.805134008969e+02 
>>>>   0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
>>>> 28 SNES Function norm 2.805133530790e+02 
>>>>   0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
>>>> 29 SNES Function norm 2.805133509342e+02 
>>>>   0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
>>>> 30 SNES Function norm 2.805133508006e+02 
>>>>   0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
>>>> 31 SNES Function norm 2.805133508001e+02 
>>>>   0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
>>>> 32 SNES Function norm 2.805133507963e+02 
>>>>   0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> Please run with the -snes_fd and one level of grid sequencing (where Newton was having trouble converging on the second level) with -snes_monitor -ksp_monitor_true_residual and send the output. Too much speculation and not enough data.
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/1fe30815/attachment-0001.html>

From bsmith at mcs.anl.gov  Wed Sep  9 22:40:18 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 9 Sep 2015 22:40:18 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <E6733B83-B018-4B4C-BC93-41AE8844D6DB@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
	<9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
	<CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>
	<0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>
	<FEA5A943-DD17-4B4C-AD1F-F1F1F04E95DD@mcs.anl.gov>
	<E6733B83-B018-4B4C-BC93-41AE8844D6DB@gmail.com>
Message-ID: <B97304A9-1CBF-4D9B-A5B8-8277313638A2@mcs.anl.gov>


  The lines that include "near zero implies found a local minimum"



> On Sep 9, 2015, at 10:08 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> The output is 55 megs.  Is there some specific portion you?re looking for? 
> 
> -gideon
> 
>> On Sep 9, 2015, at 6:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>   Ok, it is able to solve the linear systems with very small residuals so I don't believe the Jacobian is single at this location.
>> 
>>   Please make the same run again but with the additional option -info and send all the output.
>> 
>>  Barry
>> 
>> 
>>> On Sep 9, 2015, at 5:29 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> I did a fresh install of 3.6.1 against gcc 5.2 with ?download-mpi and ?download-f2cblas with quad math support.  I then ran my code with the solver flags:
>>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>>> 		-pc_type lu \
>>> 		-snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual -binary_read_double
>>> 
>>> I get similar results as before, but, clearly, at higher precision:
>>> 
>>>    0 SNES Function norm 2.528207888951e-12 
>>>      0 KSP preconditioned resid norm 3.518112956043e-12 true resid norm 2.528207888951e-12 ||r(i)||/||b|| 1.000000000000e+00
>>>      1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 1.904732438038e-41 ||r(i)||/||b|| 7.533923323167e-30
>>>    1 SNES Function norm 2.299665925647e-23 
>>>  Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1
>>>  0 SNES Function norm 5.066222213176e+03 
>>>    0 KSP preconditioned resid norm 3.135087050102e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.700962799224e-30 true resid norm 5.578720034265e-27 ||r(i)||/||b|| 1.101159759585e-30
>>>  1 SNES Function norm 8.484697184455e+02 
>>>    0 KSP preconditioned resid norm 2.668083368285e+02 true resid norm 8.484697184455e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 8.794084418808e-29 true resid norm 2.437641286218e-26 ||r(i)||/||b|| 2.872985603639e-29
>>>  2 SNES Function norm 6.549559723473e+02 
>>>    0 KSP preconditioned resid norm 1.552154272986e+02 true resid norm 6.549559723473e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.507070573051e-29 true resid norm 4.361978496405e-27 ||r(i)||/||b|| 6.659956822398e-30
>>>  3 SNES Function norm 5.770723278332e+02 
>>>    0 KSP preconditioned resid norm 1.348596652974e+02 true resid norm 5.770723278332e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.578833635012e-28 true resid norm 1.182025127733e-26 ||r(i)||/||b|| 2.048313652764e-29
>>>  4 SNES Function norm 5.237702240779e+02 
>>>    0 KSP preconditioned resid norm 1.524401223281e+02 true resid norm 5.237702240779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.348074578423e-29 true resid norm 3.851281508331e-27 ||r(i)||/||b|| 7.352998187537e-30
>>>  5 SNES Function norm 4.753909020039e+02 
>>>    0 KSP preconditioned resid norm 1.864261890236e+02 true resid norm 4.753909020039e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 7.878414198958e-29 true resid norm 5.419935080688e-26 ||r(i)||/||b|| 1.140100716661e-28
>>>  6 SNES Function norm 4.221784590918e+02 
>>>    0 KSP preconditioned resid norm 2.550992139992e+02 true resid norm 4.221784590918e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 4.075971594113e-29 true resid norm 7.094679472303e-27 ||r(i)||/||b|| 1.680493004680e-29
>>>  7 SNES Function norm 3.806525080588e+02 
>>>    0 KSP preconditioned resid norm 5.183605289859e+02 true resid norm 3.806525080588e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.103573513147e-28 true resid norm 2.144465704039e-26 ||r(i)||/||b|| 5.633657098375e-29
>>>  8 SNES Function norm 3.762054656159e+02 
>>>    0 KSP preconditioned resid norm 5.320776223634e+03 true resid norm 3.762054656159e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.585030728289e-27 true resid norm 5.530627313578e-25 ||r(i)||/||b|| 1.470108177329e-27
>>>  9 SNES Function norm 3.758975227011e+02 
>>>    0 KSP preconditioned resid norm 6.252711433929e+03 true resid norm 3.758975227011e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 6.414696869631e-28 true resid norm 2.875693139366e-25 ||r(i)||/||b|| 7.650205084359e-28
>>> 10 SNES Function norm 3.757032042685e+02 
>>>    0 KSP preconditioned resid norm 5.890497158679e+02 true resid norm 3.757032042685e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.912199122959e-28 true resid norm 4.765625322483e-26 ||r(i)||/||b|| 1.268454798452e-28
>>> 11 SNES Function norm 3.728798164350e+02 
>>>    0 KSP preconditioned resid norm 1.081356529363e+03 true resid norm 3.728798164350e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.465322278036e-27 true resid norm 8.948779168104e-26 ||r(i)||/||b|| 2.399909776201e-28
>>> 12 SNES Function norm 3.723078740346e+02 
>>>    0 KSP preconditioned resid norm 3.745127665225e+03 true resid norm 3.723078740346e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 6.682508949012e-28 true resid norm 2.402288617952e-26 ||r(i)||/||b|| 6.452424956579e-29
>>> 13 SNES Function norm 3.721848058457e+02 
>>>    0 KSP preconditioned resid norm 3.495477557738e+03 true resid norm 3.721848058457e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.389031769454e-28 true resid norm 4.589073952100e-26 ||r(i)||/||b|| 1.233009483467e-28
>>> 14 SNES Function norm 3.720227574913e+02 
>>>    0 KSP preconditioned resid norm 1.265999878583e+04 true resid norm 3.720227574913e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.936728612764e-27 true resid norm 6.774363900864e-26 ||r(i)||/||b|| 1.820954165962e-28
>>> 15 SNES Function norm 3.720051997704e+02 
>>>    0 KSP preconditioned resid norm 2.912354770053e+03 true resid norm 3.720051997704e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.193330450218e-26 true resid norm 9.007685186918e-25 ||r(i)||/||b|| 2.421386903322e-27
>>> 16 SNES Function norm 3.718945465838e+02 
>>>    0 KSP preconditioned resid norm 1.082057529463e+03 true resid norm 3.718945465838e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.143696537658e-28 true resid norm 1.173159164030e-26 ||r(i)||/||b|| 3.154547908288e-29
>>> 17 SNES Function norm 3.700412588801e+02 
>>>    0 KSP preconditioned resid norm 3.676198400599e+01 true resid norm 3.700412588801e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.884191202285e-29 true resid norm 8.141362804702e-28 ||r(i)||/||b|| 2.200122988810e-30
>>> 18 SNES Function norm 3.351964193518e+02 
>>>    0 KSP preconditioned resid norm 1.071494896915e+02 true resid norm 3.351964193518e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 5.159408051956e-29 true resid norm 3.785844889574e-27 ||r(i)||/||b|| 1.129440731168e-29
>>> 19 SNES Function norm 3.096020694819e+02 
>>>    0 KSP preconditioned resid norm 3.361110777562e+02 true resid norm 3.096020694819e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.067163043170e-28 true resid norm 3.097548865490e-26 ||r(i)||/||b|| 1.000493591879e-28
>>> 20 SNES Function norm 3.008463400554e+02 
>>>    0 KSP preconditioned resid norm 7.151989169521e+01 true resid norm 3.008463400554e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.963021196462e-29 true resid norm 3.267842047327e-27 ||r(i)||/||b|| 1.086216321171e-29
>>> 21 SNES Function norm 2.752372240183e+02 
>>>    0 KSP preconditioned resid norm 1.737818994441e+02 true resid norm 2.752372240183e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.355051821355e-29 true resid norm 1.159582173983e-27 ||r(i)||/||b|| 4.213028154600e-30
>>> 22 SNES Function norm 2.707638991136e+02 
>>>    0 KSP preconditioned resid norm 4.095365990261e+02 true resid norm 2.707638991136e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.313462531085e-29 true resid norm 2.326415447553e-27 ||r(i)||/||b|| 8.592044416441e-30
>>> 23 SNES Function norm 2.698513655244e+02 
>>>    0 KSP preconditioned resid norm 7.571732975899e+01 true resid norm 2.698513655244e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 4.706642213758e-29 true resid norm 5.275659070101e-28 ||r(i)||/||b|| 1.955024040679e-30
>>> 24 SNES Function norm 2.617973287453e+02 
>>>    0 KSP preconditioned resid norm 1.781133707863e+02 true resid norm 2.617973287453e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 7.624863846438e-29 true resid norm 1.041367251217e-27 ||r(i)||/||b|| 3.977761179642e-30
>>> 25 SNES Function norm 2.600135095660e+02 
>>>    0 KSP preconditioned resid norm 1.377746106386e+02 true resid norm 2.600135095660e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.148291819547e-28 true resid norm 1.521069513901e-27 ||r(i)||/||b|| 5.849963397822e-30
>>> 26 SNES Function norm 2.584547548464e+02 
>>>    0 KSP preconditioned resid norm 1.778914250872e+02 true resid norm 2.584547548464e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.300336366195e-28 true resid norm 2.069166109298e-27 ||r(i)||/||b|| 8.005912332807e-30
>>> 27 SNES Function norm 2.570427136855e+02 
>>>    0 KSP preconditioned resid norm 9.377626460831e+01 true resid norm 2.570427136855e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 5.766637486688e-29 true resid norm 5.436682842489e-28 ||r(i)||/||b|| 2.115089264557e-30
>>> 28 SNES Function norm 2.553675835085e+02 
>>>    0 KSP preconditioned resid norm 3.196813637571e+02 true resid norm 2.553675835085e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 5.412882946976e-28 true resid norm 4.755028260087e-26 ||r(i)||/||b|| 1.862032837041e-28
>>> 29 SNES Function norm 2.543107878046e+02 
>>>    0 KSP preconditioned resid norm 3.373192219793e+02 true resid norm 2.543107878046e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.620724126195e-29 true resid norm 2.234654091065e-27 ||r(i)||/||b|| 8.787099085950e-30
>>> 30 SNES Function norm 2.539436050245e+02 
>>>    0 KSP preconditioned resid norm 3.351117240495e+02 true resid norm 2.539436050245e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.229207196291e-29 true resid norm 3.225569644851e-27 ||r(i)||/||b|| 1.270191326354e-29
>>> 31 SNES Function norm 2.536651625665e+02 
>>>    0 KSP preconditioned resid norm 3.687817755461e+02 true resid norm 2.536651625665e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.233376086959e-29 true resid norm 2.522505835235e-27 ||r(i)||/||b|| 9.944234398264e-30
>>> 32 SNES Function norm 2.536237446853e+02 
>>>    0 KSP preconditioned resid norm 6.074223711615e+02 true resid norm 2.536237446853e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.047078682230e-28 true resid norm 5.038976546196e-27 ||r(i)||/||b|| 1.986792109094e-29
>>> 33 SNES Function norm 2.535658772687e+02 
>>>    0 KSP preconditioned resid norm 8.774992658126e+02 true resid norm 2.535658772687e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.036350427238e-28 true resid norm 4.058678859635e-27 ||r(i)||/||b|| 1.600640789428e-29
>>> 34 SNES Function norm 2.535415069989e+02 
>>>    0 KSP preconditioned resid norm 1.619728779960e+03 true resid norm 2.535415069989e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.173925947581e-28 true resid norm 1.460901190689e-26 ||r(i)||/||b|| 5.761980387282e-29
>>> 35 SNES Function norm 2.535362549500e+02 
>>>    0 KSP preconditioned resid norm 4.120252504215e+03 true resid norm 2.535362549500e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.482004320762e-27 true resid norm 2.515988923022e-26 ||r(i)||/||b|| 9.923586366447e-29
>>> 36 SNES Function norm 2.535350372131e+02 
>>>    0 KSP preconditioned resid norm 7.065615798134e+03 true resid norm 2.535350372131e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.267231805260e-27 true resid norm 9.127341716497e-26 ||r(i)||/||b|| 3.600031702452e-28
>>> 37 SNES Function norm 2.535346214155e+02 
>>>    0 KSP preconditioned resid norm 1.200630987401e+04 true resid norm 2.535346214155e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 6.331744826877e-27 true resid norm 3.434648364475e-25 ||r(i)||/||b|| 1.354705856462e-27
>>> 38 SNES Function norm 2.535344771151e+02 
>>>    0 KSP preconditioned resid norm 2.045653956385e+04 true resid norm 2.535344771151e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.729474459326e-27 true resid norm 4.809771115033e-25 ||r(i)||/||b|| 1.897087595250e-27
>>> 39 SNES Function norm 2.535344273428e+02 
>>>    0 KSP preconditioned resid norm 3.496422852684e+04 true resid norm 2.535344273428e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.413460430546e-26 true resid norm 2.800309113539e-25 ||r(i)||/||b|| 1.104508426287e-27
>>> 40 SNES Function norm 2.535344102918e+02 
>>>    0 KSP preconditioned resid norm 5.986286390122e+04 true resid norm 2.535344102918e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.071099626460e-25 true resid norm 2.269549087060e-24 ||r(i)||/||b|| 8.951641256299e-27
>>> 41 SNES Function norm 2.535344046223e+02 
>>>    0 KSP preconditioned resid norm 1.158888344932e+05 true resid norm 2.535344046223e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 9.362165764120e-26 true resid norm 5.723929359721e-24 ||r(i)||/||b|| 2.257653894448e-26
>>> 42 SNES Function norm 2.535344044297e+02 
>>>    0 KSP preconditioned resid norm 6.015343562155e+05 true resid norm 2.535344044297e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 3.045273774078e-24 true resid norm 5.965631634103e-23 ||r(i)||/||b|| 2.352987022618e-25
>>> 43 SNES Function norm 2.535344043720e+02 
>>>    0 KSP preconditioned resid norm 1.032398569969e+06 true resid norm 2.535344043720e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 6.133229797301e-24 true resid norm 2.132052198244e-22 ||r(i)||/||b|| 8.409321028933e-25
>>> 44 SNES Function norm 2.535344043525e+02 
>>>    0 KSP preconditioned resid norm 1.771990642532e+06 true resid norm 2.535344043525e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 1.399736530559e-23 true resid norm 2.816123472957e-22 ||r(i)||/||b|| 1.110746085980e-24
>>> 45 SNES Function norm 2.535344043458e+02 
>>>    0 KSP preconditioned resid norm 3.041524303674e+06 true resid norm 2.535344043458e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 6.700523240131e-24 true resid norm 1.214940402127e-21 ||r(i)||/||b|| 4.792013948806e-24
>>> 46 SNES Function norm 2.535344043436e+02 
>>>    0 KSP preconditioned resid norm 5.220719804889e+06 true resid norm 2.535344043436e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 7.749472721038e-24 true resid norm 9.462448677416e-22 ||r(i)||/||b|| 3.732214845522e-24
>>> 47 SNES Function norm 2.535344043428e+02 
>>>    0 KSP preconditioned resid norm 8.961379294572e+06 true resid norm 2.535344043428e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.735576070962e-23 true resid norm 3.261102152175e-21 ||r(i)||/||b|| 1.286256261997e-23
>>> 48 SNES Function norm 2.535344043425e+02 
>>>    0 KSP preconditioned resid norm 1.538234198056e+07 true resid norm 2.535344043425e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 5.924165850746e-23 true resid norm 3.890747245316e-21 ||r(i)||/||b|| 1.534603264360e-23
>>> 49 SNES Function norm 2.535344043424e+02 
>>>    0 KSP preconditioned resid norm 2.640413180607e+07 true resid norm 2.535344043424e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>    1 KSP preconditioned resid norm 2.163013019454e-23 true resid norm 5.034396892612e-21 ||r(i)||/||b|| 1.985685889719e-23
>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 49
>>> -gideon
>>> 
>>>> On Sep 9, 2015, at 3:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>>  Thanks. I think you should try running this with __float128 (quad precision) to see what happens.
>>>> 
>>>>  Do you have access to a relatively recent (last couple of years) gcc (and gfortran) compiler?
>>>> 
>>>>  Select a new PETSC_ARCH name say arch-quad
>>>> 
>>>>  and run configure with the additional option -with-precision=__float128 
>>>> 
>>>>  not use --download-mumps or other such packages since they don't work with __float128.  If you get everything build then run
>>>> with the same options except do not use pc_factor_mat_solver_package mumps  just use the default PETSc solver LU
>>>> 
>>>>  Note that if your program is reading in binary files this will be slightly tricky since you will need to convert the result to __float128 format. If you use PETSc VecLoad() or PetscBinaryRead() etc you can just use the flag -binary_read_double and will read from a double binary file and automatically put the results into the PETSc __float128 matrices.
>>>> 
>>>> Interpreting results. You may get convergence which might likely indicate the scaling and conditioning of your problem are very nasty but can be overcome with more precision or you may get the same lack of convergence you get now and I don't have a good explanation for that :-)
>>>> 
>>>> Note: it could also be that your SNES function evaluation has a problem.
>>>> 
>>>> 
>>>> 
>>>>  Barry
>>>> 
>>>>> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> So here?s one grid sequence with these solver flags:
>>>>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>>>>> -pc_type lu \
>>>>> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1 -mat_mumps_icntl_7 6 \
>>>>> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
>>>>> 
>>>>> By the way, the problem, I believe is approaching something that is singular, so it is not unreasonable to think there is some kind of inherent ill conditioning in the problem.
>>>>> 
>>>>>   0 SNES Function norm 2.977789375068e-12 
>>>>>     0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>>>>>     1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>>>>>   1 SNES Function norm 2.511921299174e-12 
>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>> 0 SNES Function norm 5.066222213175e+03 
>>>>>   0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
>>>>> 1 SNES Function norm 8.484664416715e+02 
>>>>>   0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
>>>>> 2 SNES Function norm 6.549557200006e+02 
>>>>>   0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
>>>>> 3 SNES Function norm 5.770704424482e+02 
>>>>>   0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
>>>>> 4 SNES Function norm 5.237674957045e+02 
>>>>>   0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
>>>>> 5 SNES Function norm 4.753878819749e+02 
>>>>>   0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
>>>>> 6 SNES Function norm 4.221755368773e+02 
>>>>>   0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
>>>>> 7 SNES Function norm 3.806522663388e+02 
>>>>>   0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
>>>>> 8 SNES Function norm 3.762033603907e+02 
>>>>>   0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
>>>>> 9 SNES Function norm 3.758953286234e+02 
>>>>>   0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
>>>>> 10 SNES Function norm 3.756870388529e+02 
>>>>>   0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
>>>>> 11 SNES Function norm 3.728715884883e+02 
>>>>>   0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
>>>>> 12 SNES Function norm 3.722328158041e+02 
>>>>>   0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
>>>>> 13 SNES Function norm 3.720989356426e+02 
>>>>>   0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
>>>>> 14 SNES Function norm 3.720707403757e+02 
>>>>>   0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
>>>>> 15 SNES Function norm 3.720707293983e+02 
>>>>>   0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
>>>>> 16 SNES Function norm 3.719027845818e+02 
>>>>>   0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
>>>>> 17 SNES Function norm 3.714400762925e+02 
>>>>>   0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
>>>>> 18 SNES Function norm 3.701626928766e+02 
>>>>>   0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
>>>>> 19 SNES Function norm 3.556442798407e+02 
>>>>>   0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
>>>>> 20 SNES Function norm 3.496125058841e+02 
>>>>>   0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
>>>>> 21 SNES Function norm 3.146776071310e+02 
>>>>>   0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
>>>>> 22 SNES Function norm 2.855422195269e+02 
>>>>>   0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
>>>>> 23 SNES Function norm 2.806777486780e+02 
>>>>>   0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
>>>>> 24 SNES Function norm 2.805893466480e+02 
>>>>>   0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
>>>>> 25 SNES Function norm 2.805171735939e+02 
>>>>>   0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
>>>>> 26 SNES Function norm 2.805136061483e+02 
>>>>>   0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
>>>>> 27 SNES Function norm 2.805134008969e+02 
>>>>>   0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
>>>>> 28 SNES Function norm 2.805133530790e+02 
>>>>>   0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
>>>>> 29 SNES Function norm 2.805133509342e+02 
>>>>>   0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
>>>>> 30 SNES Function norm 2.805133508006e+02 
>>>>>   0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
>>>>> 31 SNES Function norm 2.805133508001e+02 
>>>>>   0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
>>>>> 32 SNES Function norm 2.805133507963e+02 
>>>>>   0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> Please run with the -snes_fd and one level of grid sequencing (where Newton was having trouble converging on the second level) with -snes_monitor -ksp_monitor_true_residual and send the output. Too much speculation and not enough data.
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From gideon.simpson at gmail.com  Thu Sep 10 05:27:30 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Thu, 10 Sep 2015 06:27:30 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <B97304A9-1CBF-4D9B-A5B8-8277313638A2@mcs.anl.gov>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
	<9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
	<CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>
	<0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>
	<FEA5A943-DD17-4B4C-AD1F-F1F1F04E95DD@mcs.anl.gov>
	<E6733B83-B018-4B4C-BC93-41AE8844D6DB@gmail.com>
	<B97304A9-1CBF-4D9B-A5B8-8277313638A2@mcs.anl.gov>
Message-ID: <F9D32EFA-9B44-46A0-BF71-1BB5FADB9089@gmail.com>

So, again, running with quad precision,  -snes_fd and pc_type lu, if I do a grep for ?near zero?, I see the following:

[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 8.163023801098e+03 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 3.146370982548e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.697372735008e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.823526830758e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.507319553253e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.813628310312e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.816618020961e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.817849506863e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.807060603357e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.809091235931e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.623262449138e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.816756713682e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.685357274378e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.756938620885e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.680026061303e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.683148573781e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.618467045595e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.623293118989e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.668093905256e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.695212648533e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.620604473902e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.672385534279e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.652972728605e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.698411440904e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.542965823842e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.609012612946e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.578931076912e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.663406483563e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.586963839897e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.683149398378e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.680332769742e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.756753069476e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.711014670480e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.696818088632e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.721356438844e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.683784209263e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.641393054798e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.604645386683e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.589098449105e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.584733045067e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.637110790126e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.585837492699e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.588112071344e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.585162738429e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.585159448454e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.584858718181e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.584588706770e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.583202923956e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.584796625595e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.584435892572e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.586950970155e+04 near zero implies inconsistent rhs
[0] SNESNEWTONLSCheckLocalMin_Private(): || J^T F|| 1.245983339100e+04 near zero implies found a local minimum


-gideon

> On Sep 9, 2015, at 11:40 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  The lines that include "near zero implies found a local minimum"
> 
> 
> 
>> On Sep 9, 2015, at 10:08 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> The output is 55 megs.  Is there some specific portion you?re looking for? 
>> 
>> -gideon
>> 
>>> On Sep 9, 2015, at 6:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>>  Ok, it is able to solve the linear systems with very small residuals so I don't believe the Jacobian is single at this location.
>>> 
>>>  Please make the same run again but with the additional option -info and send all the output.
>>> 
>>> Barry
>>> 
>>> 
>>>> On Sep 9, 2015, at 5:29 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> I did a fresh install of 3.6.1 against gcc 5.2 with ?download-mpi and ?download-f2cblas with quad math support.  I then ran my code with the solver flags:
>>>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>>>> 		-pc_type lu \
>>>> 		-snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual -binary_read_double
>>>> 
>>>> I get similar results as before, but, clearly, at higher precision:
>>>> 
>>>>   0 SNES Function norm 2.528207888951e-12 
>>>>     0 KSP preconditioned resid norm 3.518112956043e-12 true resid norm 2.528207888951e-12 ||r(i)||/||b|| 1.000000000000e+00
>>>>     1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 1.904732438038e-41 ||r(i)||/||b|| 7.533923323167e-30
>>>>   1 SNES Function norm 2.299665925647e-23 
>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1
>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>   0 KSP preconditioned resid norm 3.135087050102e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.700962799224e-30 true resid norm 5.578720034265e-27 ||r(i)||/||b|| 1.101159759585e-30
>>>> 1 SNES Function norm 8.484697184455e+02 
>>>>   0 KSP preconditioned resid norm 2.668083368285e+02 true resid norm 8.484697184455e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 8.794084418808e-29 true resid norm 2.437641286218e-26 ||r(i)||/||b|| 2.872985603639e-29
>>>> 2 SNES Function norm 6.549559723473e+02 
>>>>   0 KSP preconditioned resid norm 1.552154272986e+02 true resid norm 6.549559723473e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.507070573051e-29 true resid norm 4.361978496405e-27 ||r(i)||/||b|| 6.659956822398e-30
>>>> 3 SNES Function norm 5.770723278332e+02 
>>>>   0 KSP preconditioned resid norm 1.348596652974e+02 true resid norm 5.770723278332e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.578833635012e-28 true resid norm 1.182025127733e-26 ||r(i)||/||b|| 2.048313652764e-29
>>>> 4 SNES Function norm 5.237702240779e+02 
>>>>   0 KSP preconditioned resid norm 1.524401223281e+02 true resid norm 5.237702240779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.348074578423e-29 true resid norm 3.851281508331e-27 ||r(i)||/||b|| 7.352998187537e-30
>>>> 5 SNES Function norm 4.753909020039e+02 
>>>>   0 KSP preconditioned resid norm 1.864261890236e+02 true resid norm 4.753909020039e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 7.878414198958e-29 true resid norm 5.419935080688e-26 ||r(i)||/||b|| 1.140100716661e-28
>>>> 6 SNES Function norm 4.221784590918e+02 
>>>>   0 KSP preconditioned resid norm 2.550992139992e+02 true resid norm 4.221784590918e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 4.075971594113e-29 true resid norm 7.094679472303e-27 ||r(i)||/||b|| 1.680493004680e-29
>>>> 7 SNES Function norm 3.806525080588e+02 
>>>>   0 KSP preconditioned resid norm 5.183605289859e+02 true resid norm 3.806525080588e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.103573513147e-28 true resid norm 2.144465704039e-26 ||r(i)||/||b|| 5.633657098375e-29
>>>> 8 SNES Function norm 3.762054656159e+02 
>>>>   0 KSP preconditioned resid norm 5.320776223634e+03 true resid norm 3.762054656159e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.585030728289e-27 true resid norm 5.530627313578e-25 ||r(i)||/||b|| 1.470108177329e-27
>>>> 9 SNES Function norm 3.758975227011e+02 
>>>>   0 KSP preconditioned resid norm 6.252711433929e+03 true resid norm 3.758975227011e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 6.414696869631e-28 true resid norm 2.875693139366e-25 ||r(i)||/||b|| 7.650205084359e-28
>>>> 10 SNES Function norm 3.757032042685e+02 
>>>>   0 KSP preconditioned resid norm 5.890497158679e+02 true resid norm 3.757032042685e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.912199122959e-28 true resid norm 4.765625322483e-26 ||r(i)||/||b|| 1.268454798452e-28
>>>> 11 SNES Function norm 3.728798164350e+02 
>>>>   0 KSP preconditioned resid norm 1.081356529363e+03 true resid norm 3.728798164350e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.465322278036e-27 true resid norm 8.948779168104e-26 ||r(i)||/||b|| 2.399909776201e-28
>>>> 12 SNES Function norm 3.723078740346e+02 
>>>>   0 KSP preconditioned resid norm 3.745127665225e+03 true resid norm 3.723078740346e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 6.682508949012e-28 true resid norm 2.402288617952e-26 ||r(i)||/||b|| 6.452424956579e-29
>>>> 13 SNES Function norm 3.721848058457e+02 
>>>>   0 KSP preconditioned resid norm 3.495477557738e+03 true resid norm 3.721848058457e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.389031769454e-28 true resid norm 4.589073952100e-26 ||r(i)||/||b|| 1.233009483467e-28
>>>> 14 SNES Function norm 3.720227574913e+02 
>>>>   0 KSP preconditioned resid norm 1.265999878583e+04 true resid norm 3.720227574913e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.936728612764e-27 true resid norm 6.774363900864e-26 ||r(i)||/||b|| 1.820954165962e-28
>>>> 15 SNES Function norm 3.720051997704e+02 
>>>>   0 KSP preconditioned resid norm 2.912354770053e+03 true resid norm 3.720051997704e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.193330450218e-26 true resid norm 9.007685186918e-25 ||r(i)||/||b|| 2.421386903322e-27
>>>> 16 SNES Function norm 3.718945465838e+02 
>>>>   0 KSP preconditioned resid norm 1.082057529463e+03 true resid norm 3.718945465838e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.143696537658e-28 true resid norm 1.173159164030e-26 ||r(i)||/||b|| 3.154547908288e-29
>>>> 17 SNES Function norm 3.700412588801e+02 
>>>>   0 KSP preconditioned resid norm 3.676198400599e+01 true resid norm 3.700412588801e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.884191202285e-29 true resid norm 8.141362804702e-28 ||r(i)||/||b|| 2.200122988810e-30
>>>> 18 SNES Function norm 3.351964193518e+02 
>>>>   0 KSP preconditioned resid norm 1.071494896915e+02 true resid norm 3.351964193518e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 5.159408051956e-29 true resid norm 3.785844889574e-27 ||r(i)||/||b|| 1.129440731168e-29
>>>> 19 SNES Function norm 3.096020694819e+02 
>>>>   0 KSP preconditioned resid norm 3.361110777562e+02 true resid norm 3.096020694819e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.067163043170e-28 true resid norm 3.097548865490e-26 ||r(i)||/||b|| 1.000493591879e-28
>>>> 20 SNES Function norm 3.008463400554e+02 
>>>>   0 KSP preconditioned resid norm 7.151989169521e+01 true resid norm 3.008463400554e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.963021196462e-29 true resid norm 3.267842047327e-27 ||r(i)||/||b|| 1.086216321171e-29
>>>> 21 SNES Function norm 2.752372240183e+02 
>>>>   0 KSP preconditioned resid norm 1.737818994441e+02 true resid norm 2.752372240183e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.355051821355e-29 true resid norm 1.159582173983e-27 ||r(i)||/||b|| 4.213028154600e-30
>>>> 22 SNES Function norm 2.707638991136e+02 
>>>>   0 KSP preconditioned resid norm 4.095365990261e+02 true resid norm 2.707638991136e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.313462531085e-29 true resid norm 2.326415447553e-27 ||r(i)||/||b|| 8.592044416441e-30
>>>> 23 SNES Function norm 2.698513655244e+02 
>>>>   0 KSP preconditioned resid norm 7.571732975899e+01 true resid norm 2.698513655244e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 4.706642213758e-29 true resid norm 5.275659070101e-28 ||r(i)||/||b|| 1.955024040679e-30
>>>> 24 SNES Function norm 2.617973287453e+02 
>>>>   0 KSP preconditioned resid norm 1.781133707863e+02 true resid norm 2.617973287453e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 7.624863846438e-29 true resid norm 1.041367251217e-27 ||r(i)||/||b|| 3.977761179642e-30
>>>> 25 SNES Function norm 2.600135095660e+02 
>>>>   0 KSP preconditioned resid norm 1.377746106386e+02 true resid norm 2.600135095660e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.148291819547e-28 true resid norm 1.521069513901e-27 ||r(i)||/||b|| 5.849963397822e-30
>>>> 26 SNES Function norm 2.584547548464e+02 
>>>>   0 KSP preconditioned resid norm 1.778914250872e+02 true resid norm 2.584547548464e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.300336366195e-28 true resid norm 2.069166109298e-27 ||r(i)||/||b|| 8.005912332807e-30
>>>> 27 SNES Function norm 2.570427136855e+02 
>>>>   0 KSP preconditioned resid norm 9.377626460831e+01 true resid norm 2.570427136855e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 5.766637486688e-29 true resid norm 5.436682842489e-28 ||r(i)||/||b|| 2.115089264557e-30
>>>> 28 SNES Function norm 2.553675835085e+02 
>>>>   0 KSP preconditioned resid norm 3.196813637571e+02 true resid norm 2.553675835085e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 5.412882946976e-28 true resid norm 4.755028260087e-26 ||r(i)||/||b|| 1.862032837041e-28
>>>> 29 SNES Function norm 2.543107878046e+02 
>>>>   0 KSP preconditioned resid norm 3.373192219793e+02 true resid norm 2.543107878046e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.620724126195e-29 true resid norm 2.234654091065e-27 ||r(i)||/||b|| 8.787099085950e-30
>>>> 30 SNES Function norm 2.539436050245e+02 
>>>>   0 KSP preconditioned resid norm 3.351117240495e+02 true resid norm 2.539436050245e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.229207196291e-29 true resid norm 3.225569644851e-27 ||r(i)||/||b|| 1.270191326354e-29
>>>> 31 SNES Function norm 2.536651625665e+02 
>>>>   0 KSP preconditioned resid norm 3.687817755461e+02 true resid norm 2.536651625665e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.233376086959e-29 true resid norm 2.522505835235e-27 ||r(i)||/||b|| 9.944234398264e-30
>>>> 32 SNES Function norm 2.536237446853e+02 
>>>>   0 KSP preconditioned resid norm 6.074223711615e+02 true resid norm 2.536237446853e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.047078682230e-28 true resid norm 5.038976546196e-27 ||r(i)||/||b|| 1.986792109094e-29
>>>> 33 SNES Function norm 2.535658772687e+02 
>>>>   0 KSP preconditioned resid norm 8.774992658126e+02 true resid norm 2.535658772687e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.036350427238e-28 true resid norm 4.058678859635e-27 ||r(i)||/||b|| 1.600640789428e-29
>>>> 34 SNES Function norm 2.535415069989e+02 
>>>>   0 KSP preconditioned resid norm 1.619728779960e+03 true resid norm 2.535415069989e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.173925947581e-28 true resid norm 1.460901190689e-26 ||r(i)||/||b|| 5.761980387282e-29
>>>> 35 SNES Function norm 2.535362549500e+02 
>>>>   0 KSP preconditioned resid norm 4.120252504215e+03 true resid norm 2.535362549500e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.482004320762e-27 true resid norm 2.515988923022e-26 ||r(i)||/||b|| 9.923586366447e-29
>>>> 36 SNES Function norm 2.535350372131e+02 
>>>>   0 KSP preconditioned resid norm 7.065615798134e+03 true resid norm 2.535350372131e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.267231805260e-27 true resid norm 9.127341716497e-26 ||r(i)||/||b|| 3.600031702452e-28
>>>> 37 SNES Function norm 2.535346214155e+02 
>>>>   0 KSP preconditioned resid norm 1.200630987401e+04 true resid norm 2.535346214155e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 6.331744826877e-27 true resid norm 3.434648364475e-25 ||r(i)||/||b|| 1.354705856462e-27
>>>> 38 SNES Function norm 2.535344771151e+02 
>>>>   0 KSP preconditioned resid norm 2.045653956385e+04 true resid norm 2.535344771151e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.729474459326e-27 true resid norm 4.809771115033e-25 ||r(i)||/||b|| 1.897087595250e-27
>>>> 39 SNES Function norm 2.535344273428e+02 
>>>>   0 KSP preconditioned resid norm 3.496422852684e+04 true resid norm 2.535344273428e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.413460430546e-26 true resid norm 2.800309113539e-25 ||r(i)||/||b|| 1.104508426287e-27
>>>> 40 SNES Function norm 2.535344102918e+02 
>>>>   0 KSP preconditioned resid norm 5.986286390122e+04 true resid norm 2.535344102918e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.071099626460e-25 true resid norm 2.269549087060e-24 ||r(i)||/||b|| 8.951641256299e-27
>>>> 41 SNES Function norm 2.535344046223e+02 
>>>>   0 KSP preconditioned resid norm 1.158888344932e+05 true resid norm 2.535344046223e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 9.362165764120e-26 true resid norm 5.723929359721e-24 ||r(i)||/||b|| 2.257653894448e-26
>>>> 42 SNES Function norm 2.535344044297e+02 
>>>>   0 KSP preconditioned resid norm 6.015343562155e+05 true resid norm 2.535344044297e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 3.045273774078e-24 true resid norm 5.965631634103e-23 ||r(i)||/||b|| 2.352987022618e-25
>>>> 43 SNES Function norm 2.535344043720e+02 
>>>>   0 KSP preconditioned resid norm 1.032398569969e+06 true resid norm 2.535344043720e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 6.133229797301e-24 true resid norm 2.132052198244e-22 ||r(i)||/||b|| 8.409321028933e-25
>>>> 44 SNES Function norm 2.535344043525e+02 
>>>>   0 KSP preconditioned resid norm 1.771990642532e+06 true resid norm 2.535344043525e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.399736530559e-23 true resid norm 2.816123472957e-22 ||r(i)||/||b|| 1.110746085980e-24
>>>> 45 SNES Function norm 2.535344043458e+02 
>>>>   0 KSP preconditioned resid norm 3.041524303674e+06 true resid norm 2.535344043458e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 6.700523240131e-24 true resid norm 1.214940402127e-21 ||r(i)||/||b|| 4.792013948806e-24
>>>> 46 SNES Function norm 2.535344043436e+02 
>>>>   0 KSP preconditioned resid norm 5.220719804889e+06 true resid norm 2.535344043436e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 7.749472721038e-24 true resid norm 9.462448677416e-22 ||r(i)||/||b|| 3.732214845522e-24
>>>> 47 SNES Function norm 2.535344043428e+02 
>>>>   0 KSP preconditioned resid norm 8.961379294572e+06 true resid norm 2.535344043428e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.735576070962e-23 true resid norm 3.261102152175e-21 ||r(i)||/||b|| 1.286256261997e-23
>>>> 48 SNES Function norm 2.535344043425e+02 
>>>>   0 KSP preconditioned resid norm 1.538234198056e+07 true resid norm 2.535344043425e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 5.924165850746e-23 true resid norm 3.890747245316e-21 ||r(i)||/||b|| 1.534603264360e-23
>>>> 49 SNES Function norm 2.535344043424e+02 
>>>>   0 KSP preconditioned resid norm 2.640413180607e+07 true resid norm 2.535344043424e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 2.163013019454e-23 true resid norm 5.034396892612e-21 ||r(i)||/||b|| 1.985685889719e-23
>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 49
>>>> -gideon
>>>> 
>>>>> On Sep 9, 2015, at 3:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> Thanks. I think you should try running this with __float128 (quad precision) to see what happens.
>>>>> 
>>>>> Do you have access to a relatively recent (last couple of years) gcc (and gfortran) compiler?
>>>>> 
>>>>> Select a new PETSC_ARCH name say arch-quad
>>>>> 
>>>>> and run configure with the additional option -with-precision=__float128 
>>>>> 
>>>>> not use --download-mumps or other such packages since they don't work with __float128.  If you get everything build then run
>>>>> with the same options except do not use pc_factor_mat_solver_package mumps  just use the default PETSc solver LU
>>>>> 
>>>>> Note that if your program is reading in binary files this will be slightly tricky since you will need to convert the result to __float128 format. If you use PETSc VecLoad() or PetscBinaryRead() etc you can just use the flag -binary_read_double and will read from a double binary file and automatically put the results into the PETSc __float128 matrices.
>>>>> 
>>>>> Interpreting results. You may get convergence which might likely indicate the scaling and conditioning of your problem are very nasty but can be overcome with more precision or you may get the same lack of convergence you get now and I don't have a good explanation for that :-)
>>>>> 
>>>>> Note: it could also be that your SNES function evaluation has a problem.
>>>>> 
>>>>> 
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> So here?s one grid sequence with these solver flags:
>>>>>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>>>>>> -pc_type lu \
>>>>>> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1 -mat_mumps_icntl_7 6 \
>>>>>> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
>>>>>> 
>>>>>> By the way, the problem, I believe is approaching something that is singular, so it is not unreasonable to think there is some kind of inherent ill conditioning in the problem.
>>>>>> 
>>>>>>  0 SNES Function norm 2.977789375068e-12 
>>>>>>    0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>    1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>>>>>>  1 SNES Function norm 2.511921299174e-12 
>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>> 0 SNES Function norm 5.066222213175e+03 
>>>>>>  0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
>>>>>> 1 SNES Function norm 8.484664416715e+02 
>>>>>>  0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
>>>>>> 2 SNES Function norm 6.549557200006e+02 
>>>>>>  0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
>>>>>> 3 SNES Function norm 5.770704424482e+02 
>>>>>>  0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
>>>>>> 4 SNES Function norm 5.237674957045e+02 
>>>>>>  0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
>>>>>> 5 SNES Function norm 4.753878819749e+02 
>>>>>>  0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
>>>>>> 6 SNES Function norm 4.221755368773e+02 
>>>>>>  0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
>>>>>> 7 SNES Function norm 3.806522663388e+02 
>>>>>>  0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
>>>>>> 8 SNES Function norm 3.762033603907e+02 
>>>>>>  0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
>>>>>> 9 SNES Function norm 3.758953286234e+02 
>>>>>>  0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
>>>>>> 10 SNES Function norm 3.756870388529e+02 
>>>>>>  0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
>>>>>> 11 SNES Function norm 3.728715884883e+02 
>>>>>>  0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
>>>>>> 12 SNES Function norm 3.722328158041e+02 
>>>>>>  0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
>>>>>> 13 SNES Function norm 3.720989356426e+02 
>>>>>>  0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
>>>>>> 14 SNES Function norm 3.720707403757e+02 
>>>>>>  0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
>>>>>> 15 SNES Function norm 3.720707293983e+02 
>>>>>>  0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
>>>>>> 16 SNES Function norm 3.719027845818e+02 
>>>>>>  0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
>>>>>> 17 SNES Function norm 3.714400762925e+02 
>>>>>>  0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
>>>>>> 18 SNES Function norm 3.701626928766e+02 
>>>>>>  0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
>>>>>> 19 SNES Function norm 3.556442798407e+02 
>>>>>>  0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
>>>>>> 20 SNES Function norm 3.496125058841e+02 
>>>>>>  0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
>>>>>> 21 SNES Function norm 3.146776071310e+02 
>>>>>>  0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
>>>>>> 22 SNES Function norm 2.855422195269e+02 
>>>>>>  0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
>>>>>> 23 SNES Function norm 2.806777486780e+02 
>>>>>>  0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
>>>>>> 24 SNES Function norm 2.805893466480e+02 
>>>>>>  0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
>>>>>> 25 SNES Function norm 2.805171735939e+02 
>>>>>>  0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
>>>>>> 26 SNES Function norm 2.805136061483e+02 
>>>>>>  0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
>>>>>> 27 SNES Function norm 2.805134008969e+02 
>>>>>>  0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
>>>>>> 28 SNES Function norm 2.805133530790e+02 
>>>>>>  0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
>>>>>> 29 SNES Function norm 2.805133509342e+02 
>>>>>>  0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
>>>>>> 30 SNES Function norm 2.805133508006e+02 
>>>>>>  0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
>>>>>> 31 SNES Function norm 2.805133508001e+02 
>>>>>>  0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
>>>>>> 32 SNES Function norm 2.805133507963e+02 
>>>>>>  0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>  1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> Please run with the -snes_fd and one level of grid sequencing (where Newton was having trouble converging on the second level) with -snes_monitor -ksp_monitor_true_residual and send the output. Too much speculation and not enough data.
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150910/83ebc245/attachment-0001.html>

From knepley at gmail.com  Thu Sep 10 05:50:56 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 10 Sep 2015 05:50:56 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <F9D32EFA-9B44-46A0-BF71-1BB5FADB9089@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
	<9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
	<CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>
	<0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>
	<FEA5A943-DD17-4B4C-AD1F-F1F1F04E95DD@mcs.anl.gov>
	<E6733B83-B018-4B4C-BC93-41AE8844D6DB@gmail.com>
	<B97304A9-1CBF-4D9B-A5B8-8277313638A2@mcs.anl.gov>
	<F9D32EFA-9B44-46A0-BF71-1BB5FADB9089@gmail.com>
Message-ID: <CAMYG4GnV06oX1gi-nS=XbDSw=fWWQ7fYCO1C9=ktFd+LRwmmfQ@mail.gmail.com>

On Thu, Sep 10, 2015 at 5:27 AM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> So, again, running with quad precision,  -snes_fd and pc_type lu, if I do
> a grep for ?near zero?, I see the following:
>

So Newton is unjustified in crapping out. Our guess is just far from the
convergence region.

Can you show the errors norms for both halves of the problem?

  Thanks,

     Matt


> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 8.163023801098e+03 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 3.146370982548e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.697372735008e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.823526830758e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.507319553253e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.813628310312e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.816618020961e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.817849506863e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.807060603357e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.809091235931e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.623262449138e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.816756713682e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.685357274378e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.756938620885e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.680026061303e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.683148573781e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.618467045595e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.623293118989e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.668093905256e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.695212648533e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.620604473902e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.672385534279e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.652972728605e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.698411440904e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.542965823842e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.609012612946e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.578931076912e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.663406483563e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.586963839897e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.683149398378e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.680332769742e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.756753069476e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.711014670480e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.696818088632e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.721356438844e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.683784209263e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.641393054798e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.604645386683e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.589098449105e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.584733045067e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.637110790126e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.585837492699e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.588112071344e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.585162738429e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.585159448454e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.584858718181e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.584588706770e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.583202923956e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.584796625595e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.584435892572e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
> 1.586950970155e+04 near zero implies inconsistent rhs
> [0] SNESNEWTONLSCheckLocalMin_Private(): || J^T F|| 1.245983339100e+04
> near zero implies found a local minimum
>
>
> -gideon
>
> On Sep 9, 2015, at 11:40 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>  The lines that include "near zero implies found a local minimum"
>
>
>
> On Sep 9, 2015, at 10:08 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> The output is 55 megs.  Is there some specific portion you?re looking for?
>
> -gideon
>
> On Sep 9, 2015, at 6:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>  Ok, it is able to solve the linear systems with very small residuals so I
> don't believe the Jacobian is single at this location.
>
>  Please make the same run again but with the additional option -info and
> send all the output.
>
> Barry
>
>
> On Sep 9, 2015, at 5:29 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> I did a fresh install of 3.6.1 against gcc 5.2 with ?download-mpi and
> ?download-f2cblas with quad math support.  I then ran my code with the
> solver flags:
> -snes_max_it 100 -snes_converged_reason -snes_monitor \
> -pc_type lu \
> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
> -binary_read_double
>
> I get similar results as before, but, clearly, at higher precision:
>
>   0 SNES Function norm 2.528207888951e-12
>     0 KSP preconditioned resid norm 3.518112956043e-12 true resid norm
> 2.528207888951e-12 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm
> 1.904732438038e-41 ||r(i)||/||b|| 7.533923323167e-30
>   1 SNES Function norm 2.299665925647e-23
> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1
> 0 SNES Function norm 5.066222213176e+03
>   0 KSP preconditioned resid norm 3.135087050102e+01 true resid norm
> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.700962799224e-30 true resid norm
> 5.578720034265e-27 ||r(i)||/||b|| 1.101159759585e-30
> 1 SNES Function norm 8.484697184455e+02
>   0 KSP preconditioned resid norm 2.668083368285e+02 true resid norm
> 8.484697184455e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 8.794084418808e-29 true resid norm
> 2.437641286218e-26 ||r(i)||/||b|| 2.872985603639e-29
> 2 SNES Function norm 6.549559723473e+02
>   0 KSP preconditioned resid norm 1.552154272986e+02 true resid norm
> 6.549559723473e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.507070573051e-29 true resid norm
> 4.361978496405e-27 ||r(i)||/||b|| 6.659956822398e-30
> 3 SNES Function norm 5.770723278332e+02
>   0 KSP preconditioned resid norm 1.348596652974e+02 true resid norm
> 5.770723278332e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.578833635012e-28 true resid norm
> 1.182025127733e-26 ||r(i)||/||b|| 2.048313652764e-29
> 4 SNES Function norm 5.237702240779e+02
>   0 KSP preconditioned resid norm 1.524401223281e+02 true resid norm
> 5.237702240779e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.348074578423e-29 true resid norm
> 3.851281508331e-27 ||r(i)||/||b|| 7.352998187537e-30
> 5 SNES Function norm 4.753909020039e+02
>   0 KSP preconditioned resid norm 1.864261890236e+02 true resid norm
> 4.753909020039e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 7.878414198958e-29 true resid norm
> 5.419935080688e-26 ||r(i)||/||b|| 1.140100716661e-28
> 6 SNES Function norm 4.221784590918e+02
>   0 KSP preconditioned resid norm 2.550992139992e+02 true resid norm
> 4.221784590918e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 4.075971594113e-29 true resid norm
> 7.094679472303e-27 ||r(i)||/||b|| 1.680493004680e-29
> 7 SNES Function norm 3.806525080588e+02
>   0 KSP preconditioned resid norm 5.183605289859e+02 true resid norm
> 3.806525080588e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.103573513147e-28 true resid norm
> 2.144465704039e-26 ||r(i)||/||b|| 5.633657098375e-29
> 8 SNES Function norm 3.762054656159e+02
>   0 KSP preconditioned resid norm 5.320776223634e+03 true resid norm
> 3.762054656159e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.585030728289e-27 true resid norm
> 5.530627313578e-25 ||r(i)||/||b|| 1.470108177329e-27
> 9 SNES Function norm 3.758975227011e+02
>   0 KSP preconditioned resid norm 6.252711433929e+03 true resid norm
> 3.758975227011e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 6.414696869631e-28 true resid norm
> 2.875693139366e-25 ||r(i)||/||b|| 7.650205084359e-28
> 10 SNES Function norm 3.757032042685e+02
>   0 KSP preconditioned resid norm 5.890497158679e+02 true resid norm
> 3.757032042685e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.912199122959e-28 true resid norm
> 4.765625322483e-26 ||r(i)||/||b|| 1.268454798452e-28
> 11 SNES Function norm 3.728798164350e+02
>   0 KSP preconditioned resid norm 1.081356529363e+03 true resid norm
> 3.728798164350e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.465322278036e-27 true resid norm
> 8.948779168104e-26 ||r(i)||/||b|| 2.399909776201e-28
> 12 SNES Function norm 3.723078740346e+02
>   0 KSP preconditioned resid norm 3.745127665225e+03 true resid norm
> 3.723078740346e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 6.682508949012e-28 true resid norm
> 2.402288617952e-26 ||r(i)||/||b|| 6.452424956579e-29
> 13 SNES Function norm 3.721848058457e+02
>   0 KSP preconditioned resid norm 3.495477557738e+03 true resid norm
> 3.721848058457e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.389031769454e-28 true resid norm
> 4.589073952100e-26 ||r(i)||/||b|| 1.233009483467e-28
> 14 SNES Function norm 3.720227574913e+02
>   0 KSP preconditioned resid norm 1.265999878583e+04 true resid norm
> 3.720227574913e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.936728612764e-27 true resid norm
> 6.774363900864e-26 ||r(i)||/||b|| 1.820954165962e-28
> 15 SNES Function norm 3.720051997704e+02
>   0 KSP preconditioned resid norm 2.912354770053e+03 true resid norm
> 3.720051997704e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.193330450218e-26 true resid norm
> 9.007685186918e-25 ||r(i)||/||b|| 2.421386903322e-27
> 16 SNES Function norm 3.718945465838e+02
>   0 KSP preconditioned resid norm 1.082057529463e+03 true resid norm
> 3.718945465838e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.143696537658e-28 true resid norm
> 1.173159164030e-26 ||r(i)||/||b|| 3.154547908288e-29
> 17 SNES Function norm 3.700412588801e+02
>   0 KSP preconditioned resid norm 3.676198400599e+01 true resid norm
> 3.700412588801e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.884191202285e-29 true resid norm
> 8.141362804702e-28 ||r(i)||/||b|| 2.200122988810e-30
> 18 SNES Function norm 3.351964193518e+02
>   0 KSP preconditioned resid norm 1.071494896915e+02 true resid norm
> 3.351964193518e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 5.159408051956e-29 true resid norm
> 3.785844889574e-27 ||r(i)||/||b|| 1.129440731168e-29
> 19 SNES Function norm 3.096020694819e+02
>   0 KSP preconditioned resid norm 3.361110777562e+02 true resid norm
> 3.096020694819e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.067163043170e-28 true resid norm
> 3.097548865490e-26 ||r(i)||/||b|| 1.000493591879e-28
> 20 SNES Function norm 3.008463400554e+02
>   0 KSP preconditioned resid norm 7.151989169521e+01 true resid norm
> 3.008463400554e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.963021196462e-29 true resid norm
> 3.267842047327e-27 ||r(i)||/||b|| 1.086216321171e-29
> 21 SNES Function norm 2.752372240183e+02
>   0 KSP preconditioned resid norm 1.737818994441e+02 true resid norm
> 2.752372240183e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.355051821355e-29 true resid norm
> 1.159582173983e-27 ||r(i)||/||b|| 4.213028154600e-30
> 22 SNES Function norm 2.707638991136e+02
>   0 KSP preconditioned resid norm 4.095365990261e+02 true resid norm
> 2.707638991136e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.313462531085e-29 true resid norm
> 2.326415447553e-27 ||r(i)||/||b|| 8.592044416441e-30
> 23 SNES Function norm 2.698513655244e+02
>   0 KSP preconditioned resid norm 7.571732975899e+01 true resid norm
> 2.698513655244e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 4.706642213758e-29 true resid norm
> 5.275659070101e-28 ||r(i)||/||b|| 1.955024040679e-30
> 24 SNES Function norm 2.617973287453e+02
>   0 KSP preconditioned resid norm 1.781133707863e+02 true resid norm
> 2.617973287453e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 7.624863846438e-29 true resid norm
> 1.041367251217e-27 ||r(i)||/||b|| 3.977761179642e-30
> 25 SNES Function norm 2.600135095660e+02
>   0 KSP preconditioned resid norm 1.377746106386e+02 true resid norm
> 2.600135095660e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.148291819547e-28 true resid norm
> 1.521069513901e-27 ||r(i)||/||b|| 5.849963397822e-30
> 26 SNES Function norm 2.584547548464e+02
>   0 KSP preconditioned resid norm 1.778914250872e+02 true resid norm
> 2.584547548464e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.300336366195e-28 true resid norm
> 2.069166109298e-27 ||r(i)||/||b|| 8.005912332807e-30
> 27 SNES Function norm 2.570427136855e+02
>   0 KSP preconditioned resid norm 9.377626460831e+01 true resid norm
> 2.570427136855e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 5.766637486688e-29 true resid norm
> 5.436682842489e-28 ||r(i)||/||b|| 2.115089264557e-30
> 28 SNES Function norm 2.553675835085e+02
>   0 KSP preconditioned resid norm 3.196813637571e+02 true resid norm
> 2.553675835085e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 5.412882946976e-28 true resid norm
> 4.755028260087e-26 ||r(i)||/||b|| 1.862032837041e-28
> 29 SNES Function norm 2.543107878046e+02
>   0 KSP preconditioned resid norm 3.373192219793e+02 true resid norm
> 2.543107878046e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.620724126195e-29 true resid norm
> 2.234654091065e-27 ||r(i)||/||b|| 8.787099085950e-30
> 30 SNES Function norm 2.539436050245e+02
>   0 KSP preconditioned resid norm 3.351117240495e+02 true resid norm
> 2.539436050245e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.229207196291e-29 true resid norm
> 3.225569644851e-27 ||r(i)||/||b|| 1.270191326354e-29
> 31 SNES Function norm 2.536651625665e+02
>   0 KSP preconditioned resid norm 3.687817755461e+02 true resid norm
> 2.536651625665e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.233376086959e-29 true resid norm
> 2.522505835235e-27 ||r(i)||/||b|| 9.944234398264e-30
> 32 SNES Function norm 2.536237446853e+02
>   0 KSP preconditioned resid norm 6.074223711615e+02 true resid norm
> 2.536237446853e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.047078682230e-28 true resid norm
> 5.038976546196e-27 ||r(i)||/||b|| 1.986792109094e-29
> 33 SNES Function norm 2.535658772687e+02
>   0 KSP preconditioned resid norm 8.774992658126e+02 true resid norm
> 2.535658772687e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.036350427238e-28 true resid norm
> 4.058678859635e-27 ||r(i)||/||b|| 1.600640789428e-29
> 34 SNES Function norm 2.535415069989e+02
>   0 KSP preconditioned resid norm 1.619728779960e+03 true resid norm
> 2.535415069989e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.173925947581e-28 true resid norm
> 1.460901190689e-26 ||r(i)||/||b|| 5.761980387282e-29
> 35 SNES Function norm 2.535362549500e+02
>   0 KSP preconditioned resid norm 4.120252504215e+03 true resid norm
> 2.535362549500e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.482004320762e-27 true resid norm
> 2.515988923022e-26 ||r(i)||/||b|| 9.923586366447e-29
> 36 SNES Function norm 2.535350372131e+02
>   0 KSP preconditioned resid norm 7.065615798134e+03 true resid norm
> 2.535350372131e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.267231805260e-27 true resid norm
> 9.127341716497e-26 ||r(i)||/||b|| 3.600031702452e-28
> 37 SNES Function norm 2.535346214155e+02
>   0 KSP preconditioned resid norm 1.200630987401e+04 true resid norm
> 2.535346214155e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 6.331744826877e-27 true resid norm
> 3.434648364475e-25 ||r(i)||/||b|| 1.354705856462e-27
> 38 SNES Function norm 2.535344771151e+02
>   0 KSP preconditioned resid norm 2.045653956385e+04 true resid norm
> 2.535344771151e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.729474459326e-27 true resid norm
> 4.809771115033e-25 ||r(i)||/||b|| 1.897087595250e-27
> 39 SNES Function norm 2.535344273428e+02
>   0 KSP preconditioned resid norm 3.496422852684e+04 true resid norm
> 2.535344273428e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.413460430546e-26 true resid norm
> 2.800309113539e-25 ||r(i)||/||b|| 1.104508426287e-27
> 40 SNES Function norm 2.535344102918e+02
>   0 KSP preconditioned resid norm 5.986286390122e+04 true resid norm
> 2.535344102918e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.071099626460e-25 true resid norm
> 2.269549087060e-24 ||r(i)||/||b|| 8.951641256299e-27
> 41 SNES Function norm 2.535344046223e+02
>   0 KSP preconditioned resid norm 1.158888344932e+05 true resid norm
> 2.535344046223e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 9.362165764120e-26 true resid norm
> 5.723929359721e-24 ||r(i)||/||b|| 2.257653894448e-26
> 42 SNES Function norm 2.535344044297e+02
>   0 KSP preconditioned resid norm 6.015343562155e+05 true resid norm
> 2.535344044297e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 3.045273774078e-24 true resid norm
> 5.965631634103e-23 ||r(i)||/||b|| 2.352987022618e-25
> 43 SNES Function norm 2.535344043720e+02
>   0 KSP preconditioned resid norm 1.032398569969e+06 true resid norm
> 2.535344043720e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 6.133229797301e-24 true resid norm
> 2.132052198244e-22 ||r(i)||/||b|| 8.409321028933e-25
> 44 SNES Function norm 2.535344043525e+02
>   0 KSP preconditioned resid norm 1.771990642532e+06 true resid norm
> 2.535344043525e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.399736530559e-23 true resid norm
> 2.816123472957e-22 ||r(i)||/||b|| 1.110746085980e-24
> 45 SNES Function norm 2.535344043458e+02
>   0 KSP preconditioned resid norm 3.041524303674e+06 true resid norm
> 2.535344043458e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 6.700523240131e-24 true resid norm
> 1.214940402127e-21 ||r(i)||/||b|| 4.792013948806e-24
> 46 SNES Function norm 2.535344043436e+02
>   0 KSP preconditioned resid norm 5.220719804889e+06 true resid norm
> 2.535344043436e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 7.749472721038e-24 true resid norm
> 9.462448677416e-22 ||r(i)||/||b|| 3.732214845522e-24
> 47 SNES Function norm 2.535344043428e+02
>   0 KSP preconditioned resid norm 8.961379294572e+06 true resid norm
> 2.535344043428e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.735576070962e-23 true resid norm
> 3.261102152175e-21 ||r(i)||/||b|| 1.286256261997e-23
> 48 SNES Function norm 2.535344043425e+02
>   0 KSP preconditioned resid norm 1.538234198056e+07 true resid norm
> 2.535344043425e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 5.924165850746e-23 true resid norm
> 3.890747245316e-21 ||r(i)||/||b|| 1.534603264360e-23
> 49 SNES Function norm 2.535344043424e+02
>   0 KSP preconditioned resid norm 2.640413180607e+07 true resid norm
> 2.535344043424e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.163013019454e-23 true resid norm
> 5.034396892612e-21 ||r(i)||/||b|| 1.985685889719e-23
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 49
> -gideon
>
> On Sep 9, 2015, at 3:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Thanks. I think you should try running this with __float128 (quad
> precision) to see what happens.
>
> Do you have access to a relatively recent (last couple of years) gcc (and
> gfortran) compiler?
>
> Select a new PETSC_ARCH name say arch-quad
>
> and run configure with the additional option -with-precision=__float128
>
> not use --download-mumps or other such packages since they don't work with
> __float128.  If you get everything build then run
> with the same options except do not use pc_factor_mat_solver_package mumps
>  just use the default PETSc solver LU
>
> Note that if your program is reading in binary files this will be slightly
> tricky since you will need to convert the result to __float128 format. If
> you use PETSc VecLoad() or PetscBinaryRead() etc you can just use the flag
> -binary_read_double and will read from a double binary file and
> automatically put the results into the PETSc __float128 matrices.
>
> Interpreting results. You may get convergence which might likely indicate
> the scaling and conditioning of your problem are very nasty but can be
> overcome with more precision or you may get the same lack of convergence
> you get now and I don't have a good explanation for that :-)
>
> Note: it could also be that your SNES function evaluation has a problem.
>
>
>
> Barry
>
> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> So here?s one grid sequence with these solver flags:
> -snes_max_it 100 -snes_converged_reason -snes_monitor \
> -pc_type lu \
> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1
> -mat_mumps_icntl_7 6 \
> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
>
> By the way, the problem, I believe is approaching something that is
> singular, so it is not unreasonable to think there is some kind of inherent
> ill conditioning in the problem.
>
>  0 SNES Function norm 2.977789375068e-12
>    0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm
> 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>    1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm
> 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>  1 SNES Function norm 2.511921299174e-12
> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
> 0 SNES Function norm 5.066222213175e+03
>  0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm
> 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm
> 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
> 1 SNES Function norm 8.484664416715e+02
>  0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm
> 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm
> 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
> 2 SNES Function norm 6.549557200006e+02
>  0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm
> 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm
> 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
> 3 SNES Function norm 5.770704424482e+02
>  0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm
> 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm
> 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
> 4 SNES Function norm 5.237674957045e+02
>  0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm
> 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm
> 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
> 5 SNES Function norm 4.753878819749e+02
>  0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm
> 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm
> 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
> 6 SNES Function norm 4.221755368773e+02
>  0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm
> 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm
> 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
> 7 SNES Function norm 3.806522663388e+02
>  0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm
> 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm
> 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
> 8 SNES Function norm 3.762033603907e+02
>  0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm
> 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm
> 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
> 9 SNES Function norm 3.758953286234e+02
>  0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm
> 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm
> 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
> 10 SNES Function norm 3.756870388529e+02
>  0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm
> 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm
> 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
> 11 SNES Function norm 3.728715884883e+02
>  0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm
> 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm
> 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
> 12 SNES Function norm 3.722328158041e+02
>  0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm
> 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm
> 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
> 13 SNES Function norm 3.720989356426e+02
>  0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm
> 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm
> 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
> 14 SNES Function norm 3.720707403757e+02
>  0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm
> 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm
> 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
> 15 SNES Function norm 3.720707293983e+02
>  0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm
> 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm
> 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
> 16 SNES Function norm 3.719027845818e+02
>  0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm
> 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm
> 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
> 17 SNES Function norm 3.714400762925e+02
>  0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm
> 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm
> 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
> 18 SNES Function norm 3.701626928766e+02
>  0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm
> 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm
> 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
> 19 SNES Function norm 3.556442798407e+02
>  0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm
> 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm
> 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
> 20 SNES Function norm 3.496125058841e+02
>  0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm
> 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm
> 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
> 21 SNES Function norm 3.146776071310e+02
>  0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm
> 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm
> 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
> 22 SNES Function norm 2.855422195269e+02
>  0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm
> 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm
> 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
> 23 SNES Function norm 2.806777486780e+02
>  0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm
> 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm
> 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
> 24 SNES Function norm 2.805893466480e+02
>  0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm
> 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm
> 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
> 25 SNES Function norm 2.805171735939e+02
>  0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm
> 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm
> 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
> 26 SNES Function norm 2.805136061483e+02
>  0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm
> 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm
> 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
> 27 SNES Function norm 2.805134008969e+02
>  0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm
> 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm
> 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
> 28 SNES Function norm 2.805133530790e+02
>  0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm
> 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm
> 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
> 29 SNES Function norm 2.805133509342e+02
>  0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm
> 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm
> 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
> 30 SNES Function norm 2.805133508006e+02
>  0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm
> 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm
> 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
> 31 SNES Function norm 2.805133508001e+02
>  0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm
> 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm
> 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
> 32 SNES Function norm 2.805133507963e+02
>  0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm
> 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm
> 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
>
> -gideon
>
> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Please run with the -snes_fd and one level of grid sequencing (where
> Newton was having trouble converging on the second level) with
> -snes_monitor -ksp_monitor_true_residual and send the output. Too much
> speculation and not enough data.
>
>
>
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150910/7b36c4b9/attachment-0001.html>

From gideon.simpson at gmail.com  Thu Sep 10 09:27:11 2015
From: gideon.simpson at gmail.com (gideon.simpson at gmail.com)
Date: Thu, 10 Sep 2015 10:27:11 -0400
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <CAMYG4GnV06oX1gi-nS=XbDSw=fWWQ7fYCO1C9=ktFd+LRwmmfQ@mail.gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
	<9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
	<CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>
	<0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>
	<FEA5A943-DD17-4B4C-AD1F-F1F1F04E95DD@mcs.anl.gov>
	<E6733B83-B018-4B4C-BC93-41AE8844D6DB@gmail.com>
	<B97304A9-1CBF-4D9B-A5B8-8277313638A2@mcs.anl.gov>
	<F9D32EFA-9B44-46A0-BF71-1BB5FADB9089@gmail.com>
	<CAMYG4GnV06oX1gi-nS=XbDSw=fWWQ7fYCO1C9=ktFd+LRwmmfQ@mail.gmail.com>
Message-ID: <3C173977-5CD2-4AEC-B6C2-E1436168E28F@gmail.com>

Should I just take the terminal vector after running the solve, evaluate it on the form function, and then get the norms of the two subvectors, or is there a more elegant approach?  Also, should we start a new thread on this?



> On Sep 10, 2015, at 6:50 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
>> On Thu, Sep 10, 2015 at 5:27 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> So, again, running with quad precision,  -snes_fd and pc_type lu, if I do a grep for ?near zero?, I see the following:
> 
> So Newton is unjustified in crapping out. Our guess is just far from the convergence region.
> 
> Can you show the errors norms for both halves of the problem?
> 
>   Thanks,
> 
>      Matt
>  
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 8.163023801098e+03 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 3.146370982548e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.697372735008e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.823526830758e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.507319553253e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.813628310312e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.816618020961e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.817849506863e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.807060603357e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.809091235931e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.623262449138e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.816756713682e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.685357274378e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.756938620885e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.680026061303e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.683148573781e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.618467045595e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.623293118989e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.668093905256e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.695212648533e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.620604473902e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.672385534279e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.652972728605e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.698411440904e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.542965823842e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.609012612946e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.578931076912e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.663406483563e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.586963839897e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.683149398378e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.680332769742e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.756753069476e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.711014670480e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.696818088632e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.721356438844e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.683784209263e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.641393054798e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.604645386683e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.589098449105e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.584733045067e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.637110790126e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.585837492699e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.588112071344e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.585162738429e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.585159448454e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.584858718181e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.584588706770e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.583202923956e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.584796625595e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.584435892572e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.586950970155e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckLocalMin_Private(): || J^T F|| 1.245983339100e+04 near zero implies found a local minimum
>> 
>> 
>> -gideon
>> 
>>> On Sep 9, 2015, at 11:40 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>>  The lines that include "near zero implies found a local minimum"
>>> 
>>> 
>>> 
>>>> On Sep 9, 2015, at 10:08 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> The output is 55 megs.  Is there some specific portion you?re looking for? 
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 9, 2015, at 6:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>  Ok, it is able to solve the linear systems with very small residuals so I don't believe the Jacobian is single at this location.
>>>>> 
>>>>>  Please make the same run again but with the additional option -info and send all the output.
>>>>> 
>>>>> Barry
>>>>> 
>>>>> 
>>>>>> On Sep 9, 2015, at 5:29 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> I did a fresh install of 3.6.1 against gcc 5.2 with ?download-mpi and ?download-f2cblas with quad math support.  I then ran my code with the solver flags:
>>>>>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>>>>>> 		-pc_type lu \
>>>>>> 		-snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual -binary_read_double
>>>>>> 
>>>>>> I get similar results as before, but, clearly, at higher precision:
>>>>>> 
>>>>>>   0 SNES Function norm 2.528207888951e-12 
>>>>>>     0 KSP preconditioned resid norm 3.518112956043e-12 true resid norm 2.528207888951e-12 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>     1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 1.904732438038e-41 ||r(i)||/||b|| 7.533923323167e-30
>>>>>>   1 SNES Function norm 2.299665925647e-23 
>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1
>>>>>> 0 SNES Function norm 5.066222213176e+03 
>>>>>>   0 KSP preconditioned resid norm 3.135087050102e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.700962799224e-30 true resid norm 5.578720034265e-27 ||r(i)||/||b|| 1.101159759585e-30
>>>>>> 1 SNES Function norm 8.484697184455e+02 
>>>>>>   0 KSP preconditioned resid norm 2.668083368285e+02 true resid norm 8.484697184455e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 8.794084418808e-29 true resid norm 2.437641286218e-26 ||r(i)||/||b|| 2.872985603639e-29
>>>>>> 2 SNES Function norm 6.549559723473e+02 
>>>>>>   0 KSP preconditioned resid norm 1.552154272986e+02 true resid norm 6.549559723473e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.507070573051e-29 true resid norm 4.361978496405e-27 ||r(i)||/||b|| 6.659956822398e-30
>>>>>> 3 SNES Function norm 5.770723278332e+02 
>>>>>>   0 KSP preconditioned resid norm 1.348596652974e+02 true resid norm 5.770723278332e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.578833635012e-28 true resid norm 1.182025127733e-26 ||r(i)||/||b|| 2.048313652764e-29
>>>>>> 4 SNES Function norm 5.237702240779e+02 
>>>>>>   0 KSP preconditioned resid norm 1.524401223281e+02 true resid norm 5.237702240779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.348074578423e-29 true resid norm 3.851281508331e-27 ||r(i)||/||b|| 7.352998187537e-30
>>>>>> 5 SNES Function norm 4.753909020039e+02 
>>>>>>   0 KSP preconditioned resid norm 1.864261890236e+02 true resid norm 4.753909020039e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 7.878414198958e-29 true resid norm 5.419935080688e-26 ||r(i)||/||b|| 1.140100716661e-28
>>>>>> 6 SNES Function norm 4.221784590918e+02 
>>>>>>   0 KSP preconditioned resid norm 2.550992139992e+02 true resid norm 4.221784590918e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 4.075971594113e-29 true resid norm 7.094679472303e-27 ||r(i)||/||b|| 1.680493004680e-29
>>>>>> 7 SNES Function norm 3.806525080588e+02 
>>>>>>   0 KSP preconditioned resid norm 5.183605289859e+02 true resid norm 3.806525080588e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.103573513147e-28 true resid norm 2.144465704039e-26 ||r(i)||/||b|| 5.633657098375e-29
>>>>>> 8 SNES Function norm 3.762054656159e+02 
>>>>>>   0 KSP preconditioned resid norm 5.320776223634e+03 true resid norm 3.762054656159e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.585030728289e-27 true resid norm 5.530627313578e-25 ||r(i)||/||b|| 1.470108177329e-27
>>>>>> 9 SNES Function norm 3.758975227011e+02 
>>>>>>   0 KSP preconditioned resid norm 6.252711433929e+03 true resid norm 3.758975227011e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 6.414696869631e-28 true resid norm 2.875693139366e-25 ||r(i)||/||b|| 7.650205084359e-28
>>>>>> 10 SNES Function norm 3.757032042685e+02 
>>>>>>   0 KSP preconditioned resid norm 5.890497158679e+02 true resid norm 3.757032042685e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.912199122959e-28 true resid norm 4.765625322483e-26 ||r(i)||/||b|| 1.268454798452e-28
>>>>>> 11 SNES Function norm 3.728798164350e+02 
>>>>>>   0 KSP preconditioned resid norm 1.081356529363e+03 true resid norm 3.728798164350e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.465322278036e-27 true resid norm 8.948779168104e-26 ||r(i)||/||b|| 2.399909776201e-28
>>>>>> 12 SNES Function norm 3.723078740346e+02 
>>>>>>   0 KSP preconditioned resid norm 3.745127665225e+03 true resid norm 3.723078740346e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 6.682508949012e-28 true resid norm 2.402288617952e-26 ||r(i)||/||b|| 6.452424956579e-29
>>>>>> 13 SNES Function norm 3.721848058457e+02 
>>>>>>   0 KSP preconditioned resid norm 3.495477557738e+03 true resid norm 3.721848058457e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.389031769454e-28 true resid norm 4.589073952100e-26 ||r(i)||/||b|| 1.233009483467e-28
>>>>>> 14 SNES Function norm 3.720227574913e+02 
>>>>>>   0 KSP preconditioned resid norm 1.265999878583e+04 true resid norm 3.720227574913e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.936728612764e-27 true resid norm 6.774363900864e-26 ||r(i)||/||b|| 1.820954165962e-28
>>>>>> 15 SNES Function norm 3.720051997704e+02 
>>>>>>   0 KSP preconditioned resid norm 2.912354770053e+03 true resid norm 3.720051997704e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.193330450218e-26 true resid norm 9.007685186918e-25 ||r(i)||/||b|| 2.421386903322e-27
>>>>>> 16 SNES Function norm 3.718945465838e+02 
>>>>>>   0 KSP preconditioned resid norm 1.082057529463e+03 true resid norm 3.718945465838e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.143696537658e-28 true resid norm 1.173159164030e-26 ||r(i)||/||b|| 3.154547908288e-29
>>>>>> 17 SNES Function norm 3.700412588801e+02 
>>>>>>   0 KSP preconditioned resid norm 3.676198400599e+01 true resid norm 3.700412588801e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.884191202285e-29 true resid norm 8.141362804702e-28 ||r(i)||/||b|| 2.200122988810e-30
>>>>>> 18 SNES Function norm 3.351964193518e+02 
>>>>>>   0 KSP preconditioned resid norm 1.071494896915e+02 true resid norm 3.351964193518e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 5.159408051956e-29 true resid norm 3.785844889574e-27 ||r(i)||/||b|| 1.129440731168e-29
>>>>>> 19 SNES Function norm 3.096020694819e+02 
>>>>>>   0 KSP preconditioned resid norm 3.361110777562e+02 true resid norm 3.096020694819e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.067163043170e-28 true resid norm 3.097548865490e-26 ||r(i)||/||b|| 1.000493591879e-28
>>>>>> 20 SNES Function norm 3.008463400554e+02 
>>>>>>   0 KSP preconditioned resid norm 7.151989169521e+01 true resid norm 3.008463400554e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.963021196462e-29 true resid norm 3.267842047327e-27 ||r(i)||/||b|| 1.086216321171e-29
>>>>>> 21 SNES Function norm 2.752372240183e+02 
>>>>>>   0 KSP preconditioned resid norm 1.737818994441e+02 true resid norm 2.752372240183e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.355051821355e-29 true resid norm 1.159582173983e-27 ||r(i)||/||b|| 4.213028154600e-30
>>>>>> 22 SNES Function norm 2.707638991136e+02 
>>>>>>   0 KSP preconditioned resid norm 4.095365990261e+02 true resid norm 2.707638991136e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.313462531085e-29 true resid norm 2.326415447553e-27 ||r(i)||/||b|| 8.592044416441e-30
>>>>>> 23 SNES Function norm 2.698513655244e+02 
>>>>>>   0 KSP preconditioned resid norm 7.571732975899e+01 true resid norm 2.698513655244e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 4.706642213758e-29 true resid norm 5.275659070101e-28 ||r(i)||/||b|| 1.955024040679e-30
>>>>>> 24 SNES Function norm 2.617973287453e+02 
>>>>>>   0 KSP preconditioned resid norm 1.781133707863e+02 true resid norm 2.617973287453e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 7.624863846438e-29 true resid norm 1.041367251217e-27 ||r(i)||/||b|| 3.977761179642e-30
>>>>>> 25 SNES Function norm 2.600135095660e+02 
>>>>>>   0 KSP preconditioned resid norm 1.377746106386e+02 true resid norm 2.600135095660e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.148291819547e-28 true resid norm 1.521069513901e-27 ||r(i)||/||b|| 5.849963397822e-30
>>>>>> 26 SNES Function norm 2.584547548464e+02 
>>>>>>   0 KSP preconditioned resid norm 1.778914250872e+02 true resid norm 2.584547548464e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.300336366195e-28 true resid norm 2.069166109298e-27 ||r(i)||/||b|| 8.005912332807e-30
>>>>>> 27 SNES Function norm 2.570427136855e+02 
>>>>>>   0 KSP preconditioned resid norm 9.377626460831e+01 true resid norm 2.570427136855e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 5.766637486688e-29 true resid norm 5.436682842489e-28 ||r(i)||/||b|| 2.115089264557e-30
>>>>>> 28 SNES Function norm 2.553675835085e+02 
>>>>>>   0 KSP preconditioned resid norm 3.196813637571e+02 true resid norm 2.553675835085e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 5.412882946976e-28 true resid norm 4.755028260087e-26 ||r(i)||/||b|| 1.862032837041e-28
>>>>>> 29 SNES Function norm 2.543107878046e+02 
>>>>>>   0 KSP preconditioned resid norm 3.373192219793e+02 true resid norm 2.543107878046e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.620724126195e-29 true resid norm 2.234654091065e-27 ||r(i)||/||b|| 8.787099085950e-30
>>>>>> 30 SNES Function norm 2.539436050245e+02 
>>>>>>   0 KSP preconditioned resid norm 3.351117240495e+02 true resid norm 2.539436050245e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.229207196291e-29 true resid norm 3.225569644851e-27 ||r(i)||/||b|| 1.270191326354e-29
>>>>>> 31 SNES Function norm 2.536651625665e+02 
>>>>>>   0 KSP preconditioned resid norm 3.687817755461e+02 true resid norm 2.536651625665e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.233376086959e-29 true resid norm 2.522505835235e-27 ||r(i)||/||b|| 9.944234398264e-30
>>>>>> 32 SNES Function norm 2.536237446853e+02 
>>>>>>   0 KSP preconditioned resid norm 6.074223711615e+02 true resid norm 2.536237446853e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.047078682230e-28 true resid norm 5.038976546196e-27 ||r(i)||/||b|| 1.986792109094e-29
>>>>>> 33 SNES Function norm 2.535658772687e+02 
>>>>>>   0 KSP preconditioned resid norm 8.774992658126e+02 true resid norm 2.535658772687e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.036350427238e-28 true resid norm 4.058678859635e-27 ||r(i)||/||b|| 1.600640789428e-29
>>>>>> 34 SNES Function norm 2.535415069989e+02 
>>>>>>   0 KSP preconditioned resid norm 1.619728779960e+03 true resid norm 2.535415069989e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.173925947581e-28 true resid norm 1.460901190689e-26 ||r(i)||/||b|| 5.761980387282e-29
>>>>>> 35 SNES Function norm 2.535362549500e+02 
>>>>>>   0 KSP preconditioned resid norm 4.120252504215e+03 true resid norm 2.535362549500e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.482004320762e-27 true resid norm 2.515988923022e-26 ||r(i)||/||b|| 9.923586366447e-29
>>>>>> 36 SNES Function norm 2.535350372131e+02 
>>>>>>   0 KSP preconditioned resid norm 7.065615798134e+03 true resid norm 2.535350372131e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.267231805260e-27 true resid norm 9.127341716497e-26 ||r(i)||/||b|| 3.600031702452e-28
>>>>>> 37 SNES Function norm 2.535346214155e+02 
>>>>>>   0 KSP preconditioned resid norm 1.200630987401e+04 true resid norm 2.535346214155e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 6.331744826877e-27 true resid norm 3.434648364475e-25 ||r(i)||/||b|| 1.354705856462e-27
>>>>>> 38 SNES Function norm 2.535344771151e+02 
>>>>>>   0 KSP preconditioned resid norm 2.045653956385e+04 true resid norm 2.535344771151e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.729474459326e-27 true resid norm 4.809771115033e-25 ||r(i)||/||b|| 1.897087595250e-27
>>>>>> 39 SNES Function norm 2.535344273428e+02 
>>>>>>   0 KSP preconditioned resid norm 3.496422852684e+04 true resid norm 2.535344273428e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.413460430546e-26 true resid norm 2.800309113539e-25 ||r(i)||/||b|| 1.104508426287e-27
>>>>>> 40 SNES Function norm 2.535344102918e+02 
>>>>>>   0 KSP preconditioned resid norm 5.986286390122e+04 true resid norm 2.535344102918e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.071099626460e-25 true resid norm 2.269549087060e-24 ||r(i)||/||b|| 8.951641256299e-27
>>>>>> 41 SNES Function norm 2.535344046223e+02 
>>>>>>   0 KSP preconditioned resid norm 1.158888344932e+05 true resid norm 2.535344046223e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 9.362165764120e-26 true resid norm 5.723929359721e-24 ||r(i)||/||b|| 2.257653894448e-26
>>>>>> 42 SNES Function norm 2.535344044297e+02 
>>>>>>   0 KSP preconditioned resid norm 6.015343562155e+05 true resid norm 2.535344044297e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 3.045273774078e-24 true resid norm 5.965631634103e-23 ||r(i)||/||b|| 2.352987022618e-25
>>>>>> 43 SNES Function norm 2.535344043720e+02 
>>>>>>   0 KSP preconditioned resid norm 1.032398569969e+06 true resid norm 2.535344043720e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 6.133229797301e-24 true resid norm 2.132052198244e-22 ||r(i)||/||b|| 8.409321028933e-25
>>>>>> 44 SNES Function norm 2.535344043525e+02 
>>>>>>   0 KSP preconditioned resid norm 1.771990642532e+06 true resid norm 2.535344043525e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 1.399736530559e-23 true resid norm 2.816123472957e-22 ||r(i)||/||b|| 1.110746085980e-24
>>>>>> 45 SNES Function norm 2.535344043458e+02 
>>>>>>   0 KSP preconditioned resid norm 3.041524303674e+06 true resid norm 2.535344043458e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 6.700523240131e-24 true resid norm 1.214940402127e-21 ||r(i)||/||b|| 4.792013948806e-24
>>>>>> 46 SNES Function norm 2.535344043436e+02 
>>>>>>   0 KSP preconditioned resid norm 5.220719804889e+06 true resid norm 2.535344043436e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 7.749472721038e-24 true resid norm 9.462448677416e-22 ||r(i)||/||b|| 3.732214845522e-24
>>>>>> 47 SNES Function norm 2.535344043428e+02 
>>>>>>   0 KSP preconditioned resid norm 8.961379294572e+06 true resid norm 2.535344043428e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.735576070962e-23 true resid norm 3.261102152175e-21 ||r(i)||/||b|| 1.286256261997e-23
>>>>>> 48 SNES Function norm 2.535344043425e+02 
>>>>>>   0 KSP preconditioned resid norm 1.538234198056e+07 true resid norm 2.535344043425e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 5.924165850746e-23 true resid norm 3.890747245316e-21 ||r(i)||/||b|| 1.534603264360e-23
>>>>>> 49 SNES Function norm 2.535344043424e+02 
>>>>>>   0 KSP preconditioned resid norm 2.640413180607e+07 true resid norm 2.535344043424e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>   1 KSP preconditioned resid norm 2.163013019454e-23 true resid norm 5.034396892612e-21 ||r(i)||/||b|| 1.985685889719e-23
>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 49
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 9, 2015, at 3:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks. I think you should try running this with __float128 (quad precision) to see what happens.
>>>>>>> 
>>>>>>> Do you have access to a relatively recent (last couple of years) gcc (and gfortran) compiler?
>>>>>>> 
>>>>>>> Select a new PETSC_ARCH name say arch-quad
>>>>>>> 
>>>>>>> and run configure with the additional option -with-precision=__float128 
>>>>>>> 
>>>>>>> not use --download-mumps or other such packages since they don't work with __float128.  If you get everything build then run
>>>>>>> with the same options except do not use pc_factor_mat_solver_package mumps  just use the default PETSc solver LU
>>>>>>> 
>>>>>>> Note that if your program is reading in binary files this will be slightly tricky since you will need to convert the result to __float128 format. If you use PETSc VecLoad() or PetscBinaryRead() etc you can just use the flag -binary_read_double and will read from a double binary file and automatically put the results into the PETSc __float128 matrices.
>>>>>>> 
>>>>>>> Interpreting results. You may get convergence which might likely indicate the scaling and conditioning of your problem are very nasty but can be overcome with more precision or you may get the same lack of convergence you get now and I don't have a good explanation for that :-)
>>>>>>> 
>>>>>>> Note: it could also be that your SNES function evaluation has a problem.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>>> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> So here?s one grid sequence with these solver flags:
>>>>>>>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>>>>>>>> -pc_type lu \
>>>>>>>> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1 -mat_mumps_icntl_7 6 \
>>>>>>>> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
>>>>>>>> 
>>>>>>>> By the way, the problem, I believe is approaching something that is singular, so it is not unreasonable to think there is some kind of inherent ill conditioning in the problem.
>>>>>>>> 
>>>>>>>>  0 SNES Function norm 2.977789375068e-12 
>>>>>>>>    0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>    1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>>>>>>>>  1 SNES Function norm 2.511921299174e-12 
>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>> 0 SNES Function norm 5.066222213175e+03 
>>>>>>>>  0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
>>>>>>>> 1 SNES Function norm 8.484664416715e+02 
>>>>>>>>  0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
>>>>>>>> 2 SNES Function norm 6.549557200006e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
>>>>>>>> 3 SNES Function norm 5.770704424482e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
>>>>>>>> 4 SNES Function norm 5.237674957045e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
>>>>>>>> 5 SNES Function norm 4.753878819749e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
>>>>>>>> 6 SNES Function norm 4.221755368773e+02 
>>>>>>>>  0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
>>>>>>>> 7 SNES Function norm 3.806522663388e+02 
>>>>>>>>  0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
>>>>>>>> 8 SNES Function norm 3.762033603907e+02 
>>>>>>>>  0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
>>>>>>>> 9 SNES Function norm 3.758953286234e+02 
>>>>>>>>  0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
>>>>>>>> 10 SNES Function norm 3.756870388529e+02 
>>>>>>>>  0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
>>>>>>>> 11 SNES Function norm 3.728715884883e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
>>>>>>>> 12 SNES Function norm 3.722328158041e+02 
>>>>>>>>  0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
>>>>>>>> 13 SNES Function norm 3.720989356426e+02 
>>>>>>>>  0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
>>>>>>>> 14 SNES Function norm 3.720707403757e+02 
>>>>>>>>  0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
>>>>>>>> 15 SNES Function norm 3.720707293983e+02 
>>>>>>>>  0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
>>>>>>>> 16 SNES Function norm 3.719027845818e+02 
>>>>>>>>  0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
>>>>>>>> 17 SNES Function norm 3.714400762925e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
>>>>>>>> 18 SNES Function norm 3.701626928766e+02 
>>>>>>>>  0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
>>>>>>>> 19 SNES Function norm 3.556442798407e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
>>>>>>>> 20 SNES Function norm 3.496125058841e+02 
>>>>>>>>  0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
>>>>>>>> 21 SNES Function norm 3.146776071310e+02 
>>>>>>>>  0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
>>>>>>>> 22 SNES Function norm 2.855422195269e+02 
>>>>>>>>  0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
>>>>>>>> 23 SNES Function norm 2.806777486780e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
>>>>>>>> 24 SNES Function norm 2.805893466480e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
>>>>>>>> 25 SNES Function norm 2.805171735939e+02 
>>>>>>>>  0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
>>>>>>>> 26 SNES Function norm 2.805136061483e+02 
>>>>>>>>  0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
>>>>>>>> 27 SNES Function norm 2.805134008969e+02 
>>>>>>>>  0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
>>>>>>>> 28 SNES Function norm 2.805133530790e+02 
>>>>>>>>  0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
>>>>>>>> 29 SNES Function norm 2.805133509342e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
>>>>>>>> 30 SNES Function norm 2.805133508006e+02 
>>>>>>>>  0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
>>>>>>>> 31 SNES Function norm 2.805133508001e+02 
>>>>>>>>  0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
>>>>>>>> 32 SNES Function norm 2.805133507963e+02 
>>>>>>>>  0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>  1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>>>> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Please run with the -snes_fd and one level of grid sequencing (where Newton was having trouble converging on the second level) with -snes_monitor -ksp_monitor_true_residual and send the output. Too much speculation and not enough data.
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150910/d77a99e4/attachment-0001.html>

From hzhang at mcs.anl.gov  Thu Sep 10 09:44:07 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Thu, 10 Sep 2015 09:44:07 -0500
Subject: [petsc-users] Parallel 3d decomposed FFT
In-Reply-To: <55F0655F.8010803@uci.edu>
References: <55F0655F.8010803@uci.edu>
Message-ID: <CAGCphBsB9PVDJ-QrG82k5=rmryC76FMqgMOnviZy51mu7a1iKA@mail.gmail.com>

frank:

>
> I am trying to use petsc's Interface to do a 3d FFT.
> I created a 3d DMDA.  When I tried to call VecScatterPetsctoFFTW, I got
> error message saying "FFTW does not support parallel 3d FFT."

I understand that FFTW only supports 2d decomposition, like a 2d pencil.
> But is there any way I can use petsc to implement a 3d decomposed FFT?
>

I guess you do parallel 3D real transform, which is not supported by FFTW.
We are not experts on FFT. You have to search external packages that
implement it.

Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150910/6e4fbdbe/attachment.html>

From knepley at gmail.com  Thu Sep 10 09:45:30 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 10 Sep 2015 09:45:30 -0500
Subject: [petsc-users] pcfieldsplit for a composite dm with multiple
	subfields
In-Reply-To: <3C173977-5CD2-4AEC-B6C2-E1436168E28F@gmail.com>
References: <E7684D1E-FA5F-4A78-8205-F13AF5F98D8D@gmail.com>
	<E4888BDC-96F1-436E-85CF-57B196B777ED@mcs.anl.gov>
	<22066404-50E3-4BBF-9D27-26384815571A@gmail.com>
	<CAMYG4G=Trdw0_5MC33+PcXKRCGDWP7P2i9Jh07j_-mbSU8xavg@mail.gmail.com>
	<8376EFA7-D775-4D65-9E86-2303FA7E47E2@gmail.com>
	<4136AE9B-AE00-4E39-8012-888BBF920548@mcs.anl.gov>
	<C47F0A60-2B80-4FB0-8B4B-2AD2A176CBC4@gmail.com>
	<5714BD56-097D-40B9-8AEF-41273E2B512D@mcs.anl.gov>
	<ED77EE0B-CCD5-4DB9-9DC9-F411074A49DB@gmail.com>
	<05BA24AC-4011-483C-8599-5D8EED7AFE10@mcs.anl.gov>
	<CAMYG4Gmt4MJb-7PuJ_buugTrDUYxM9TKoZsptWssYyhg_ieFtw@mail.gmail.com>
	<0DBB158F-6E6B-404A-B477-BC7D5A321F01@gmail.com>
	<122D6409-96DB-4A1A-A134-525D2CEC2F1D@mcs.anl.gov>
	<3636A1D1-9B71-4C11-8A7F-424CAB000C3D@gmail.com>
	<E9DDFD9A-E597-4B30-8F8E-829A6129BF75@mcs.anl.gov>
	<ED96A250-0D7A-42C8-B861-5613152542FA@gmail.com>
	<CAMYG4Gk0B2nm+UDjHbVjVvmrBZkRRbm=+zbxjTf6iwMqUzAtng@mail.gmail.com>
	<FA1EC0C0-4FFB-4853-8787-1EF6D016A026@gmail.com>
	<8D75833E-814F-4D91-8750-05042B003A3C@mcs.anl.gov>
	<C8D0AE56-E526-4691-AE8B-EECADBA77C0D@gmail.com>
	<EF5F6F04-1052-404C-9D1B-D1FD90770497@mcs.anl.gov>
	<09F4E5F5-D2A8-4520-AB4F-751C0EBF9299@gmail.com>
	<F0E69397-2835-4489-BE69-87A4CBA50EA1@mcs.anl.gov>
	<A59FE944-94A4-40C4-8075-90819890CAC3@gmail.com>
	<81A87827-C122-481B-970D-973C014AEA44@mcs.anl.gov>
	<CAFA56ED-B662-4186-AD77-FD87811E9072@gmail.com>
	<96E35E0B-A9F6-4648-A674-34CBB34E730F@mcs.anl.gov>
	<5F92A80E-07D8-40E0-87D3-43A5DBF96AB3@gmail.com>
	<4ECB3487-83D2-40C5-977A-413640D36EF7@mcs.anl.gov>
	<75A96F3C-3516-4050-9FA8-D1500AADD595@gmail.com>
	<30155F98-44F6-4E6D-8185-405865D9A955@mcs.anl.gov>
	<2493E316-E0C5-472F-871D-8A33808D1888@gmail.com>
	<CAMYG4G=ofQ8VJCz+VmYj6PJCSd7mUMTooCepC0hDTM2pwtqWsA@mail.gmail.com>
	<5CF9C19F-1AEF-4ABB-A995-AACC76605F19@gmail.com>
	<81F7BBD3-ED28-4CD2-B38F-7D8D2C6EB3D6@mcs.anl.gov>
	<7C1C5162-EB99-4D4F-A3E2-B008BDB41ADD@gmail.com>
	<CAMYG4GkTF2vKtneeqQU+Ad396FD1-x=dMBOyR3NvqtR4x85ZmA@mail.gmail.com>
	<9B6BCA24-51AE-46B0-B6FC-482E2D8D878C@gmail.com>
	<C60882AA-549D-4417-BD95-ECD1FF789B8F@gmail.com>
	<CAMYG4GmRn1Vxyi+=FfWNKK8_2xDcXYADO5Am4dwGYPHOYo75Hw@mail.gmail.com>
	<360A3A89-1135-4AD0-B3D5-8DBC109030A7@gmail.com>
	<CAMYG4Gn8WhSqcODngJLGEkEgdTPj-Kk3pMfnp_kQwpPc_uzqEw@mail.gmail.com>
	<78A25C70-FBF6-43BF-B1E4-927AF38A2E08@gmail.com>
	<0321AE6B-8CE7-4416-AB6D-7CDD6AB8A9EF@mcs.anl.gov>
	<ECBF98CA-0C28-4CD9-9F52-89E167258FBE@gmail.com>
	<BA677D3A-97A2-463C-A45E-DCB9455F9E74@mcs.anl.gov>
	<9C4E8FA2-0A86-46A4-BB99-9A1BCE425866@gmail.com>
	<CAB869BD-D887-4B67-8D0D-86E08D060D48@mcs.anl.gov>
	<0E269E18-000F-44F3-BA53-DD2D1C2195DC@gmail.com>
	<FEA5A943-DD17-4B4C-AD1F-F1F1F04E95DD@mcs.anl.gov>
	<E6733B83-B018-4B4C-BC93-41AE8844D6DB@gmail.com>
	<B97304A9-1CBF-4D9B-A5B8-8277313638A2@mcs.anl.gov>
	<F9D32EFA-9B44-46A0-BF71-1BB5FADB9089@gmail.com>
	<CAMYG4GnV06oX1gi-nS=XbDSw=fWWQ7fYCO1C9=ktFd+LRwmmfQ@mail.gmail.com>
	<3C173977-5CD2-4AEC-B6C2-E1436168E28F@gmail.com>
Message-ID: <CAMYG4G=sKVh-iKs-GCaf6hgmweM0k+cE=gJ3axGZqm8f3Aq3xQ@mail.gmail.com>

On Thu, Sep 10, 2015 at 9:27 AM, <gideon.simpson at gmail.com> wrote:

> Should I just take the terminal vector after running the solve, evaluate
> it on the form function, and then get the norms of the two subvectors, or
> is there a more elegant approach?  Also, should we start a new thread on
> this?
>

Yes, lets start a new thread.

To start, that plan is fine, but I want residuals and not solutions.

   Matt


> On Sep 10, 2015, at 6:50 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Thu, Sep 10, 2015 at 5:27 AM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
>> So, again, running with quad precision,  -snes_fd and pc_type lu, if I do
>> a grep for ?near zero?, I see the following:
>>
>
> So Newton is unjustified in crapping out. Our guess is just far from the
> convergence region.
>
> Can you show the errors norms for both halves of the problem?
>
>   Thanks,
>
>      Matt
>
>
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 8.163023801098e+03 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 3.146370982548e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.697372735008e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.823526830758e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.507319553253e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.813628310312e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.816618020961e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.817849506863e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.807060603357e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.809091235931e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.623262449138e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.816756713682e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.685357274378e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.756938620885e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.680026061303e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.683148573781e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.618467045595e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.623293118989e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.668093905256e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.695212648533e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.620604473902e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.672385534279e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.652972728605e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.698411440904e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.542965823842e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.609012612946e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.578931076912e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.663406483563e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.586963839897e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.683149398378e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.680332769742e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.756753069476e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.711014670480e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.696818088632e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.721356438844e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.683784209263e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.641393054798e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.604645386683e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.589098449105e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.584733045067e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.637110790126e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.585837492699e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.588112071344e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.585162738429e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.585159448454e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.584858718181e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.584588706770e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.583202923956e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.584796625595e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.584435892572e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX||
>> 1.586950970155e+04 near zero implies inconsistent rhs
>> [0] SNESNEWTONLSCheckLocalMin_Private(): || J^T F|| 1.245983339100e+04
>> near zero implies found a local minimum
>>
>>
>> -gideon
>>
>> On Sep 9, 2015, at 11:40 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>>  The lines that include "near zero implies found a local minimum"
>>
>>
>>
>> On Sep 9, 2015, at 10:08 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> The output is 55 megs.  Is there some specific portion you?re looking
>> for?
>>
>> -gideon
>>
>> On Sep 9, 2015, at 6:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>>  Ok, it is able to solve the linear systems with very small residuals so
>> I don't believe the Jacobian is single at this location.
>>
>>  Please make the same run again but with the additional option -info and
>> send all the output.
>>
>> Barry
>>
>>
>> On Sep 9, 2015, at 5:29 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> I did a fresh install of 3.6.1 against gcc 5.2 with ?download-mpi and
>> ?download-f2cblas with quad math support.  I then ran my code with the
>> solver flags:
>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>> -pc_type lu \
>> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
>> -binary_read_double
>>
>> I get similar results as before, but, clearly, at higher precision:
>>
>>   0 SNES Function norm 2.528207888951e-12
>>     0 KSP preconditioned resid norm 3.518112956043e-12 true resid norm
>> 2.528207888951e-12 ||r(i)||/||b|| 1.000000000000e+00
>>     1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm
>> 1.904732438038e-41 ||r(i)||/||b|| 7.533923323167e-30
>>   1 SNES Function norm 2.299665925647e-23
>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1
>> 0 SNES Function norm 5.066222213176e+03
>>   0 KSP preconditioned resid norm 3.135087050102e+01 true resid norm
>> 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.700962799224e-30 true resid norm
>> 5.578720034265e-27 ||r(i)||/||b|| 1.101159759585e-30
>> 1 SNES Function norm 8.484697184455e+02
>>   0 KSP preconditioned resid norm 2.668083368285e+02 true resid norm
>> 8.484697184455e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 8.794084418808e-29 true resid norm
>> 2.437641286218e-26 ||r(i)||/||b|| 2.872985603639e-29
>> 2 SNES Function norm 6.549559723473e+02
>>   0 KSP preconditioned resid norm 1.552154272986e+02 true resid norm
>> 6.549559723473e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.507070573051e-29 true resid norm
>> 4.361978496405e-27 ||r(i)||/||b|| 6.659956822398e-30
>> 3 SNES Function norm 5.770723278332e+02
>>   0 KSP preconditioned resid norm 1.348596652974e+02 true resid norm
>> 5.770723278332e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.578833635012e-28 true resid norm
>> 1.182025127733e-26 ||r(i)||/||b|| 2.048313652764e-29
>> 4 SNES Function norm 5.237702240779e+02
>>   0 KSP preconditioned resid norm 1.524401223281e+02 true resid norm
>> 5.237702240779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.348074578423e-29 true resid norm
>> 3.851281508331e-27 ||r(i)||/||b|| 7.352998187537e-30
>> 5 SNES Function norm 4.753909020039e+02
>>   0 KSP preconditioned resid norm 1.864261890236e+02 true resid norm
>> 4.753909020039e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 7.878414198958e-29 true resid norm
>> 5.419935080688e-26 ||r(i)||/||b|| 1.140100716661e-28
>> 6 SNES Function norm 4.221784590918e+02
>>   0 KSP preconditioned resid norm 2.550992139992e+02 true resid norm
>> 4.221784590918e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 4.075971594113e-29 true resid norm
>> 7.094679472303e-27 ||r(i)||/||b|| 1.680493004680e-29
>> 7 SNES Function norm 3.806525080588e+02
>>   0 KSP preconditioned resid norm 5.183605289859e+02 true resid norm
>> 3.806525080588e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.103573513147e-28 true resid norm
>> 2.144465704039e-26 ||r(i)||/||b|| 5.633657098375e-29
>> 8 SNES Function norm 3.762054656159e+02
>>   0 KSP preconditioned resid norm 5.320776223634e+03 true resid norm
>> 3.762054656159e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.585030728289e-27 true resid norm
>> 5.530627313578e-25 ||r(i)||/||b|| 1.470108177329e-27
>> 9 SNES Function norm 3.758975227011e+02
>>   0 KSP preconditioned resid norm 6.252711433929e+03 true resid norm
>> 3.758975227011e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 6.414696869631e-28 true resid norm
>> 2.875693139366e-25 ||r(i)||/||b|| 7.650205084359e-28
>> 10 SNES Function norm 3.757032042685e+02
>>   0 KSP preconditioned resid norm 5.890497158679e+02 true resid norm
>> 3.757032042685e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.912199122959e-28 true resid norm
>> 4.765625322483e-26 ||r(i)||/||b|| 1.268454798452e-28
>> 11 SNES Function norm 3.728798164350e+02
>>   0 KSP preconditioned resid norm 1.081356529363e+03 true resid norm
>> 3.728798164350e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.465322278036e-27 true resid norm
>> 8.948779168104e-26 ||r(i)||/||b|| 2.399909776201e-28
>> 12 SNES Function norm 3.723078740346e+02
>>   0 KSP preconditioned resid norm 3.745127665225e+03 true resid norm
>> 3.723078740346e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 6.682508949012e-28 true resid norm
>> 2.402288617952e-26 ||r(i)||/||b|| 6.452424956579e-29
>> 13 SNES Function norm 3.721848058457e+02
>>   0 KSP preconditioned resid norm 3.495477557738e+03 true resid norm
>> 3.721848058457e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.389031769454e-28 true resid norm
>> 4.589073952100e-26 ||r(i)||/||b|| 1.233009483467e-28
>> 14 SNES Function norm 3.720227574913e+02
>>   0 KSP preconditioned resid norm 1.265999878583e+04 true resid norm
>> 3.720227574913e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.936728612764e-27 true resid norm
>> 6.774363900864e-26 ||r(i)||/||b|| 1.820954165962e-28
>> 15 SNES Function norm 3.720051997704e+02
>>   0 KSP preconditioned resid norm 2.912354770053e+03 true resid norm
>> 3.720051997704e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.193330450218e-26 true resid norm
>> 9.007685186918e-25 ||r(i)||/||b|| 2.421386903322e-27
>> 16 SNES Function norm 3.718945465838e+02
>>   0 KSP preconditioned resid norm 1.082057529463e+03 true resid norm
>> 3.718945465838e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.143696537658e-28 true resid norm
>> 1.173159164030e-26 ||r(i)||/||b|| 3.154547908288e-29
>> 17 SNES Function norm 3.700412588801e+02
>>   0 KSP preconditioned resid norm 3.676198400599e+01 true resid norm
>> 3.700412588801e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.884191202285e-29 true resid norm
>> 8.141362804702e-28 ||r(i)||/||b|| 2.200122988810e-30
>> 18 SNES Function norm 3.351964193518e+02
>>   0 KSP preconditioned resid norm 1.071494896915e+02 true resid norm
>> 3.351964193518e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 5.159408051956e-29 true resid norm
>> 3.785844889574e-27 ||r(i)||/||b|| 1.129440731168e-29
>> 19 SNES Function norm 3.096020694819e+02
>>   0 KSP preconditioned resid norm 3.361110777562e+02 true resid norm
>> 3.096020694819e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.067163043170e-28 true resid norm
>> 3.097548865490e-26 ||r(i)||/||b|| 1.000493591879e-28
>> 20 SNES Function norm 3.008463400554e+02
>>   0 KSP preconditioned resid norm 7.151989169521e+01 true resid norm
>> 3.008463400554e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.963021196462e-29 true resid norm
>> 3.267842047327e-27 ||r(i)||/||b|| 1.086216321171e-29
>> 21 SNES Function norm 2.752372240183e+02
>>   0 KSP preconditioned resid norm 1.737818994441e+02 true resid norm
>> 2.752372240183e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.355051821355e-29 true resid norm
>> 1.159582173983e-27 ||r(i)||/||b|| 4.213028154600e-30
>> 22 SNES Function norm 2.707638991136e+02
>>   0 KSP preconditioned resid norm 4.095365990261e+02 true resid norm
>> 2.707638991136e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.313462531085e-29 true resid norm
>> 2.326415447553e-27 ||r(i)||/||b|| 8.592044416441e-30
>> 23 SNES Function norm 2.698513655244e+02
>>   0 KSP preconditioned resid norm 7.571732975899e+01 true resid norm
>> 2.698513655244e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 4.706642213758e-29 true resid norm
>> 5.275659070101e-28 ||r(i)||/||b|| 1.955024040679e-30
>> 24 SNES Function norm 2.617973287453e+02
>>   0 KSP preconditioned resid norm 1.781133707863e+02 true resid norm
>> 2.617973287453e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 7.624863846438e-29 true resid norm
>> 1.041367251217e-27 ||r(i)||/||b|| 3.977761179642e-30
>> 25 SNES Function norm 2.600135095660e+02
>>   0 KSP preconditioned resid norm 1.377746106386e+02 true resid norm
>> 2.600135095660e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.148291819547e-28 true resid norm
>> 1.521069513901e-27 ||r(i)||/||b|| 5.849963397822e-30
>> 26 SNES Function norm 2.584547548464e+02
>>   0 KSP preconditioned resid norm 1.778914250872e+02 true resid norm
>> 2.584547548464e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.300336366195e-28 true resid norm
>> 2.069166109298e-27 ||r(i)||/||b|| 8.005912332807e-30
>> 27 SNES Function norm 2.570427136855e+02
>>   0 KSP preconditioned resid norm 9.377626460831e+01 true resid norm
>> 2.570427136855e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 5.766637486688e-29 true resid norm
>> 5.436682842489e-28 ||r(i)||/||b|| 2.115089264557e-30
>> 28 SNES Function norm 2.553675835085e+02
>>   0 KSP preconditioned resid norm 3.196813637571e+02 true resid norm
>> 2.553675835085e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 5.412882946976e-28 true resid norm
>> 4.755028260087e-26 ||r(i)||/||b|| 1.862032837041e-28
>> 29 SNES Function norm 2.543107878046e+02
>>   0 KSP preconditioned resid norm 3.373192219793e+02 true resid norm
>> 2.543107878046e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.620724126195e-29 true resid norm
>> 2.234654091065e-27 ||r(i)||/||b|| 8.787099085950e-30
>> 30 SNES Function norm 2.539436050245e+02
>>   0 KSP preconditioned resid norm 3.351117240495e+02 true resid norm
>> 2.539436050245e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.229207196291e-29 true resid norm
>> 3.225569644851e-27 ||r(i)||/||b|| 1.270191326354e-29
>> 31 SNES Function norm 2.536651625665e+02
>>   0 KSP preconditioned resid norm 3.687817755461e+02 true resid norm
>> 2.536651625665e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.233376086959e-29 true resid norm
>> 2.522505835235e-27 ||r(i)||/||b|| 9.944234398264e-30
>> 32 SNES Function norm 2.536237446853e+02
>>   0 KSP preconditioned resid norm 6.074223711615e+02 true resid norm
>> 2.536237446853e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.047078682230e-28 true resid norm
>> 5.038976546196e-27 ||r(i)||/||b|| 1.986792109094e-29
>> 33 SNES Function norm 2.535658772687e+02
>>   0 KSP preconditioned resid norm 8.774992658126e+02 true resid norm
>> 2.535658772687e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.036350427238e-28 true resid norm
>> 4.058678859635e-27 ||r(i)||/||b|| 1.600640789428e-29
>> 34 SNES Function norm 2.535415069989e+02
>>   0 KSP preconditioned resid norm 1.619728779960e+03 true resid norm
>> 2.535415069989e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.173925947581e-28 true resid norm
>> 1.460901190689e-26 ||r(i)||/||b|| 5.761980387282e-29
>> 35 SNES Function norm 2.535362549500e+02
>>   0 KSP preconditioned resid norm 4.120252504215e+03 true resid norm
>> 2.535362549500e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.482004320762e-27 true resid norm
>> 2.515988923022e-26 ||r(i)||/||b|| 9.923586366447e-29
>> 36 SNES Function norm 2.535350372131e+02
>>   0 KSP preconditioned resid norm 7.065615798134e+03 true resid norm
>> 2.535350372131e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.267231805260e-27 true resid norm
>> 9.127341716497e-26 ||r(i)||/||b|| 3.600031702452e-28
>> 37 SNES Function norm 2.535346214155e+02
>>   0 KSP preconditioned resid norm 1.200630987401e+04 true resid norm
>> 2.535346214155e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 6.331744826877e-27 true resid norm
>> 3.434648364475e-25 ||r(i)||/||b|| 1.354705856462e-27
>> 38 SNES Function norm 2.535344771151e+02
>>   0 KSP preconditioned resid norm 2.045653956385e+04 true resid norm
>> 2.535344771151e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.729474459326e-27 true resid norm
>> 4.809771115033e-25 ||r(i)||/||b|| 1.897087595250e-27
>> 39 SNES Function norm 2.535344273428e+02
>>   0 KSP preconditioned resid norm 3.496422852684e+04 true resid norm
>> 2.535344273428e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.413460430546e-26 true resid norm
>> 2.800309113539e-25 ||r(i)||/||b|| 1.104508426287e-27
>> 40 SNES Function norm 2.535344102918e+02
>>   0 KSP preconditioned resid norm 5.986286390122e+04 true resid norm
>> 2.535344102918e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.071099626460e-25 true resid norm
>> 2.269549087060e-24 ||r(i)||/||b|| 8.951641256299e-27
>> 41 SNES Function norm 2.535344046223e+02
>>   0 KSP preconditioned resid norm 1.158888344932e+05 true resid norm
>> 2.535344046223e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 9.362165764120e-26 true resid norm
>> 5.723929359721e-24 ||r(i)||/||b|| 2.257653894448e-26
>> 42 SNES Function norm 2.535344044297e+02
>>   0 KSP preconditioned resid norm 6.015343562155e+05 true resid norm
>> 2.535344044297e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 3.045273774078e-24 true resid norm
>> 5.965631634103e-23 ||r(i)||/||b|| 2.352987022618e-25
>> 43 SNES Function norm 2.535344043720e+02
>>   0 KSP preconditioned resid norm 1.032398569969e+06 true resid norm
>> 2.535344043720e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 6.133229797301e-24 true resid norm
>> 2.132052198244e-22 ||r(i)||/||b|| 8.409321028933e-25
>> 44 SNES Function norm 2.535344043525e+02
>>   0 KSP preconditioned resid norm 1.771990642532e+06 true resid norm
>> 2.535344043525e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.399736530559e-23 true resid norm
>> 2.816123472957e-22 ||r(i)||/||b|| 1.110746085980e-24
>> 45 SNES Function norm 2.535344043458e+02
>>   0 KSP preconditioned resid norm 3.041524303674e+06 true resid norm
>> 2.535344043458e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 6.700523240131e-24 true resid norm
>> 1.214940402127e-21 ||r(i)||/||b|| 4.792013948806e-24
>> 46 SNES Function norm 2.535344043436e+02
>>   0 KSP preconditioned resid norm 5.220719804889e+06 true resid norm
>> 2.535344043436e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 7.749472721038e-24 true resid norm
>> 9.462448677416e-22 ||r(i)||/||b|| 3.732214845522e-24
>> 47 SNES Function norm 2.535344043428e+02
>>   0 KSP preconditioned resid norm 8.961379294572e+06 true resid norm
>> 2.535344043428e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.735576070962e-23 true resid norm
>> 3.261102152175e-21 ||r(i)||/||b|| 1.286256261997e-23
>> 48 SNES Function norm 2.535344043425e+02
>>   0 KSP preconditioned resid norm 1.538234198056e+07 true resid norm
>> 2.535344043425e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 5.924165850746e-23 true resid norm
>> 3.890747245316e-21 ||r(i)||/||b|| 1.534603264360e-23
>> 49 SNES Function norm 2.535344043424e+02
>>   0 KSP preconditioned resid norm 2.640413180607e+07 true resid norm
>> 2.535344043424e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 2.163013019454e-23 true resid norm
>> 5.034396892612e-21 ||r(i)||/||b|| 1.985685889719e-23
>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 49
>> -gideon
>>
>> On Sep 9, 2015, at 3:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> Thanks. I think you should try running this with __float128 (quad
>> precision) to see what happens.
>>
>> Do you have access to a relatively recent (last couple of years) gcc (and
>> gfortran) compiler?
>>
>> Select a new PETSC_ARCH name say arch-quad
>>
>> and run configure with the additional option -with-precision=__float128
>>
>> not use --download-mumps or other such packages since they don't work
>> with __float128.  If you get everything build then run
>> with the same options except do not use pc_factor_mat_solver_package
>> mumps  just use the default PETSc solver LU
>>
>> Note that if your program is reading in binary files this will be
>> slightly tricky since you will need to convert the result to __float128
>> format. If you use PETSc VecLoad() or PetscBinaryRead() etc you can just
>> use the flag -binary_read_double and will read from a double binary file
>> and automatically put the results into the PETSc __float128 matrices.
>>
>> Interpreting results. You may get convergence which might likely indicate
>> the scaling and conditioning of your problem are very nasty but can be
>> overcome with more precision or you may get the same lack of convergence
>> you get now and I don't have a good explanation for that :-)
>>
>> Note: it could also be that your SNES function evaluation has a problem.
>>
>>
>>
>> Barry
>>
>> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> So here?s one grid sequence with these solver flags:
>> -snes_max_it 100 -snes_converged_reason -snes_monitor \
>> -pc_type lu \
>> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1
>> -mat_mumps_icntl_7 6 \
>> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
>>
>> By the way, the problem, I believe is approaching something that is
>> singular, so it is not unreasonable to think there is some kind of inherent
>> ill conditioning in the problem.
>>
>>  0 SNES Function norm 2.977789375068e-12
>>    0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm
>> 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>>    1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm
>> 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>>  1 SNES Function norm 2.511921299174e-12
>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>> 0 SNES Function norm 5.066222213175e+03
>>  0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm
>> 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm
>> 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
>> 1 SNES Function norm 8.484664416715e+02
>>  0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm
>> 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm
>> 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
>> 2 SNES Function norm 6.549557200006e+02
>>  0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm
>> 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm
>> 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
>> 3 SNES Function norm 5.770704424482e+02
>>  0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm
>> 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm
>> 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
>> 4 SNES Function norm 5.237674957045e+02
>>  0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm
>> 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm
>> 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
>> 5 SNES Function norm 4.753878819749e+02
>>  0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm
>> 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm
>> 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
>> 6 SNES Function norm 4.221755368773e+02
>>  0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm
>> 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm
>> 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
>> 7 SNES Function norm 3.806522663388e+02
>>  0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm
>> 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm
>> 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
>> 8 SNES Function norm 3.762033603907e+02
>>  0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm
>> 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm
>> 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
>> 9 SNES Function norm 3.758953286234e+02
>>  0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm
>> 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm
>> 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
>> 10 SNES Function norm 3.756870388529e+02
>>  0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm
>> 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm
>> 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
>> 11 SNES Function norm 3.728715884883e+02
>>  0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm
>> 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm
>> 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
>> 12 SNES Function norm 3.722328158041e+02
>>  0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm
>> 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm
>> 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
>> 13 SNES Function norm 3.720989356426e+02
>>  0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm
>> 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm
>> 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
>> 14 SNES Function norm 3.720707403757e+02
>>  0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm
>> 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm
>> 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
>> 15 SNES Function norm 3.720707293983e+02
>>  0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm
>> 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm
>> 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
>> 16 SNES Function norm 3.719027845818e+02
>>  0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm
>> 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm
>> 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
>> 17 SNES Function norm 3.714400762925e+02
>>  0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm
>> 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm
>> 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
>> 18 SNES Function norm 3.701626928766e+02
>>  0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm
>> 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm
>> 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
>> 19 SNES Function norm 3.556442798407e+02
>>  0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm
>> 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm
>> 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
>> 20 SNES Function norm 3.496125058841e+02
>>  0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm
>> 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm
>> 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
>> 21 SNES Function norm 3.146776071310e+02
>>  0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm
>> 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm
>> 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
>> 22 SNES Function norm 2.855422195269e+02
>>  0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm
>> 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm
>> 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
>> 23 SNES Function norm 2.806777486780e+02
>>  0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm
>> 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm
>> 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
>> 24 SNES Function norm 2.805893466480e+02
>>  0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm
>> 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm
>> 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
>> 25 SNES Function norm 2.805171735939e+02
>>  0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm
>> 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm
>> 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
>> 26 SNES Function norm 2.805136061483e+02
>>  0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm
>> 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm
>> 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
>> 27 SNES Function norm 2.805134008969e+02
>>  0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm
>> 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm
>> 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
>> 28 SNES Function norm 2.805133530790e+02
>>  0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm
>> 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm
>> 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
>> 29 SNES Function norm 2.805133509342e+02
>>  0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm
>> 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm
>> 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
>> 30 SNES Function norm 2.805133508006e+02
>>  0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm
>> 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm
>> 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
>> 31 SNES Function norm 2.805133508001e+02
>>  0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm
>> 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm
>> 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
>> 32 SNES Function norm 2.805133507963e+02
>>  0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm
>> 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm
>> 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
>>
>> -gideon
>>
>> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> Please run with the -snes_fd and one level of grid sequencing (where
>> Newton was having trouble converging on the second level) with
>> -snes_monitor -ksp_monitor_true_residual and send the output. Too much
>> speculation and not enough data.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150910/1d09cf08/attachment-0001.html>

From gideon.simpson at gmail.com  Thu Sep 10 10:40:22 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Thu, 10 Sep 2015 11:40:22 -0400
Subject: [petsc-users] snesgetfunction calling sequence
Message-ID: <1847CE38-098D-437D-A1BE-3BC907217BD5@gmail.com>

If I want to get the residual vector from a SNES, what is the correct calling order?

Currently, I have

    VecDuplicate(U,&F);


  SNESSetFunction(snes, F, form_function, ctx);

  SNESSolve(snes, NULL, U);

SNESGetFunction(snes,&F,NULL,NULL);
VecDestroy(&U);
VecDestroy(&F);

But this generates an error, 

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Wrong type of object: Parameter # 1
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
[0]PETSC ERROR: ./blowup_refine on a arch-darwin-c-debug named gs_air by gideon Thu Sep 10 11:36:10 2015
[0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
[0]PETSC ERROR: #1 VecDestroy() line 424 in /opt/petsc/src/vec/vec/interface/vector.c



-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150910/af1121bc/attachment.html>

From bsmith at mcs.anl.gov  Thu Sep 10 11:27:17 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 10 Sep 2015 11:27:17 -0500
Subject: [petsc-users] snesgetfunction calling sequence
In-Reply-To: <1847CE38-098D-437D-A1BE-3BC907217BD5@gmail.com>
References: <1847CE38-098D-437D-A1BE-3BC907217BD5@gmail.com>
Message-ID: <21E1F310-6BE4-4EED-AD1C-0CDA57162FAD@mcs.anl.gov>


   From what you've shown the code looks reasonable, but there must be something missing. Easiest thing is to run in the debugger and put a breakpoint in after the SNESGetFunction() then check what is in F.

  Barry

> On Sep 10, 2015, at 10:40 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> If I want to get the residual vector from a SNES, what is the correct calling order?
> 
> Currently, I have
> 
>     VecDuplicate(U,&F);
> 
> 
>   SNESSetFunction(snes, F, form_function, ctx);
> 
>   SNESSolve(snes, NULL, U);
> 
> SNESGetFunction(snes,&F,NULL,NULL);
> VecDestroy(&U);
> VecDestroy(&F);
> 
> But this generates an error, 
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Wrong type of object: Parameter # 1
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
> [0]PETSC ERROR: ./blowup_refine on a arch-darwin-c-debug named gs_air by gideon Thu Sep 10 11:36:10 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #1 VecDestroy() line 424 in /opt/petsc/src/vec/vec/interface/vector.c
> 
> 
> 
> -gideon
> 


From knepley at gmail.com  Thu Sep 10 11:44:36 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 10 Sep 2015 11:44:36 -0500
Subject: [petsc-users] snesgetfunction calling sequence
In-Reply-To: <21E1F310-6BE4-4EED-AD1C-0CDA57162FAD@mcs.anl.gov>
References: <1847CE38-098D-437D-A1BE-3BC907217BD5@gmail.com>
	<21E1F310-6BE4-4EED-AD1C-0CDA57162FAD@mcs.anl.gov>
Message-ID: <CAMYG4GnrG-inrhADsD9x8ze=CZGCT_K3PpbTC9aOX4bXcbFezg@mail.gmail.com>

On Thu, Sep 10, 2015 at 11:27 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    From what you've shown the code looks reasonable, but there must be
> something missing. Easiest thing is to run in the debugger and put a
> breakpoint in after the SNESGetFunction() then check what is in F


1) Do not destroy the Vec you get from SNESGEtFunction()

2) I usually prefer to re-evaluate the residual with some solution

  SNESComputeFunction()

  Thanks,

     Matt


>
>
  Barry
>
> > On Sep 10, 2015, at 10:40 AM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> >
> > If I want to get the residual vector from a SNES, what is the correct
> calling order?
> >
> > Currently, I have
> >
> >     VecDuplicate(U,&F);
> >
> >
> >   SNESSetFunction(snes, F, form_function, ctx);
> >
> >   SNESSolve(snes, NULL, U);
> >
> > SNESGetFunction(snes,&F,NULL,NULL);
> > VecDestroy(&U);
> > VecDestroy(&F);
> >
> > But this generates an error,
> >
> > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [0]PETSC ERROR: Invalid argument
> > [0]PETSC ERROR: Wrong type of object: Parameter # 1
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown
> > [0]PETSC ERROR: ./blowup_refine on a arch-darwin-c-debug named gs_air by
> gideon Thu Sep 10 11:36:10 2015
> > [0]PETSC ERROR: Configure options --download-mpich=yes
> --download-suitesparse=yes --download-superlu=yes
> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes
> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes
> --download-parmetis=yes --download-scalapack=yes
> > [0]PETSC ERROR: #1 VecDestroy() line 424 in
> /opt/petsc/src/vec/vec/interface/vector.c
> >
> >
> >
> > -gideon
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150910/d5d9002c/attachment.html>

From hypre-support at llnl.gov  Thu Sep 10 12:33:39 2015
From: hypre-support at llnl.gov (Ulrike Yang hypre Tracker)
Date: Thu, 10 Sep 2015 17:33:39 -0000
Subject: [petsc-users] [issue1287] divide by zero coming in hypre
Message-ID: <1441906419.79.0.5819217221.issue1287@llnl.gov>


Ulrike Yang <umyang at llnl.gov> added the comment:

This issue has been fixed and was tested by Barry with a prerelease.

----------
status: chatting -> resolved

_____________________________________________
hypre Issue Tracker <hypre-support at llnl.gov>
<http://tux273.llnl.gov:8070/hypre/issue1287>
_____________________________________________

From gideon.simpson at gmail.com  Thu Sep 10 15:47:18 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Thu, 10 Sep 2015 16:47:18 -0400
Subject: [petsc-users] grid sequencing & DMComposite
Message-ID: <68220B18-FFD0-473A-B87F-2C0FC1F279C3@gmail.com>

I?m using a DMCompsosite along with grid sequencing and here?s what I?m trying to accomplish.  After running the SNES solve, I?d like to evaluate the residual on the refined grid, on each piece of the DMComposite.  How do I get a DM for the refined grid which I can then use with DMCompositeGetLocalVectors in order to get each piece of the problem?  Or is there a better way?

-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150910/8b8e6a51/attachment.html>

From bsmith at mcs.anl.gov  Thu Sep 10 16:28:23 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 10 Sep 2015 16:28:23 -0500
Subject: [petsc-users] grid sequencing & DMComposite
In-Reply-To: <68220B18-FFD0-473A-B87F-2C0FC1F279C3@gmail.com>
References: <68220B18-FFD0-473A-B87F-2C0FC1F279C3@gmail.com>
Message-ID: <69870B3A-E145-4902-A577-7D065025513E@mcs.anl.gov>


  SNESGetDM()  this will return not the original DM you set but the refined one.

> On Sep 10, 2015, at 3:47 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I?m using a DMCompsosite along with grid sequencing and here?s what I?m trying to accomplish.  After running the SNES solve, I?d like to evaluate the residual on the refined grid, on each piece of the DMComposite.  How do I get a DM for the refined grid which I can then use with DMCompositeGetLocalVectors in order to get each piece of the problem?  Or is there a better way?
> 
> -gideon
> 


From jychang48 at gmail.com  Thu Sep 10 17:49:25 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Thu, 10 Sep 2015 16:49:25 -0600
Subject: [petsc-users] Expressing this nonlinear system in PETSc and/or
	petsc4py
In-Reply-To: <CAMYG4GkhhVyUSWzRnwWYAG93UuWPUR-q_1-=QsZXe15pS3oV+Q@mail.gmail.com>
References: <CAP2=TMiNLT7UK+MGGaRdmYwzvjEN4_h=hgO-5qMhjh-QOrtmYA@mail.gmail.com>
	<CAMYG4GkhhVyUSWzRnwWYAG93UuWPUR-q_1-=QsZXe15pS3oV+Q@mail.gmail.com>
Message-ID: <CAP2=TMj86cQARWvSB6C9N-+57JtpTLvrnkiWbN2BRjaaxqUFTQ@mail.gmail.com>

The example I have happens to be a simple system (e.g., Na^(+) + Cl^(-) <=>
NaCl), I could potentially solve a 7 component system with 4 primary and 3
secondary species which in some cases cannot be solved analytically as
easily.

Anyway, I was wondering if I could express the example system I had as a
mixed weak/variational form. That is, multiply each equation by a test
function and solve this system as if it were a finite element problem. I
had the following weak/variational form expressed in the FEniCS/Firedrake
form:

#==========================
mesh = ...
Q = FunctionSpace(mesh, "CG", 1)
W = Q*Q*Q
dc_A,dc_B,dc_C = TrialFunctions(W)
q_A,q_B,q_C = TestFunctions(W)
c = Function(W)
c_A,c_B,c_C = c.split()

...

F = (q_A*(u0_A - c_A - c_C) + q_B*(u0_B - c_B - c_C) + q_C*(c_C -
c_A*c_B))*dx
J = (-q_A*(dc_A + dc_C) + q_B*(dc_B + dc_C) + q_C*(dc_C - c_B*dc_A -
c_A*dc_B))*dx

...

# Solve for u0_A a u0_B

....

solve(F == 0, c, J=J)
#===========================

Where u0_A and u0_B correspond to my psi_A/B from a different function
space. However, when I solve the system, I get a DIVERGED_LINE_SEARCH. From
what I have read, it seems this error mostly arises because of an incorrect
Jacobian. But I think my Jacobian J looks correct, is it not?

Or is this even the right approach?

Thanks,
Justin


On Wed, Sep 9, 2015 at 4:41 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Sep 9, 2015 at 4:34 PM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> Hi everyone,
>>
>> I need to solve this system of nonlinear geochemical reactions:
>>
>> psi_A = c_A + c_C
>> psi_B = c_B + c_C
>> c_C = k*c_A*c_B
>>
>> where psi_A and psi_B are sub components (i.e., a mixed system) of my
>> mesh (this was done using firedrake) and k is a constant scalar. These
>> quantities are known a prior, and psi_A/B was obtained from the
>> advection-diffusion equation (using SUPG). Given the two-field formulation
>> of psi_A/B, I need to obtain a three-field formulation of c_A/B/C using the
>> above system of equations. It should be noted that the above system of
>> equations are node-independent. That is, c_A/B/C of one node does not care
>> what psi_A/B of other nodes may be.
>>
>> What?s the best strategy to go about solving this? With SciPy, i did
>> something like this:
>>
>> #====================
>> from scipy.optimize import fsolve
>> import math
>>
>> # advection-diffusion for psi_A and psi_B
>>
>> # Nonlinear function for geochemical reactions
>> def equations(p,psi_A,psi_B,k_1):
>>  c_A,c_B,c_C = p
>>  return (c_A+c_C-psi_A,c_B+c_C-psi_B,c_C-k_1*c_A*c_B)
>>
>> # Initialize
>> c_A_vec = np.zeros(len(psi_A.vector()))
>> c_B_vec = np.zeros(len(psi_A.vector()))
>> c_C_vec = np.zeros(len(psi_A.vector()))
>> k_1 = 1.0
>>
>> for i in range(len(psi_A.vector())):
>>  c_A,c_B,c_C =
>> fsolve(equations,(0.0,0.0,0.0),args=(psi_A.vector()[i],psi_B.vector()[i],k_1))
>>  c_A_vec[i] = c_A
>>  c_B_vec[i] = c_B
>>  c_C=vec[i] = c_C
>>
>> c_A = Function(Q)
>> c_B = Function(Q)
>> c_C = Function(Q)
>> c_A.vector()[:]=c_A_vec
>> c_B.vector()[:]=c_B_vec
>> c_C.vector()[:]=c_C_vec
>> #======================
>>
>> The above is my temporary work-around to this issue. Basically, what I
>> did was I solved the equations at each node. But is there a PETSc or
>> petsc4py way to do this, specifically solving the above equations globally
>> instead of at each individual node?
>>
>
> 1) I had the same thing in DFT where I had to solve a local equation for
> the screening length at each node
>
> 2) I would use a SNES
>
> 3) If you think there is variability in the solve, then do each
> individually. Otherwise,
>     you can easily do a Newton with a diagonal Jacobian.
>
> 4) I would eliminate c_C since its so easy
>
> 5) These look quadratic. Can't you solve this analytically?
>
>   Thanks,
>
>      Matt
>
>
>>
>> Thanks,
>> Justin
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150910/c846ab88/attachment.html>

From knepley at gmail.com  Fri Sep 11 06:45:41 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Sep 2015 06:45:41 -0500
Subject: [petsc-users] Expressing this nonlinear system in PETSc and/or
	petsc4py
In-Reply-To: <CAP2=TMj86cQARWvSB6C9N-+57JtpTLvrnkiWbN2BRjaaxqUFTQ@mail.gmail.com>
References: <CAP2=TMiNLT7UK+MGGaRdmYwzvjEN4_h=hgO-5qMhjh-QOrtmYA@mail.gmail.com>
	<CAMYG4GkhhVyUSWzRnwWYAG93UuWPUR-q_1-=QsZXe15pS3oV+Q@mail.gmail.com>
	<CAP2=TMj86cQARWvSB6C9N-+57JtpTLvrnkiWbN2BRjaaxqUFTQ@mail.gmail.com>
Message-ID: <CAMYG4Gk+U5xiT5HPt3+NkKAn4cRpKaND3xiS3DWfzwtt6N8png@mail.gmail.com>

On Thu, Sep 10, 2015 at 5:49 PM, Justin Chang <jychang48 at gmail.com> wrote:

> The example I have happens to be a simple system (e.g., Na^(+) + Cl^(-)
> <=> NaCl), I could potentially solve a 7 component system with 4 primary
> and 3 secondary species which in some cases cannot be solved analytically
> as easily.
>
> Anyway, I was wondering if I could express the example system I had as a
> mixed weak/variational form. That is, multiply each equation by a test
> function and solve this system as if it were a finite element problem. I
> had the following weak/variational form expressed in the FEniCS/Firedrake
> form:
>

I do not see the advantage of a variational form here. There is no "energy"
for the problem. Colocation seems easier.


> #==========================
> mesh = ...
> Q = FunctionSpace(mesh, "CG", 1)
> W = Q*Q*Q
> dc_A,dc_B,dc_C = TrialFunctions(W)
> q_A,q_B,q_C = TestFunctions(W)
> c = Function(W)
> c_A,c_B,c_C = c.split()
>
> ...
>
> F = (q_A*(u0_A - c_A - c_C) + q_B*(u0_B - c_B - c_C) + q_C*(c_C -
> c_A*c_B))*dx
> J = (-q_A*(dc_A + dc_C) + q_B*(dc_B + dc_C) + q_C*(dc_C - c_B*dc_A -
> c_A*dc_B))*dx
>
> ...
>
> # Solve for u0_A a u0_B
>
> ....
>
> solve(F == 0, c, J=J)
> #===========================
>
> Where u0_A and u0_B correspond to my psi_A/B from a different function
> space. However, when I solve the system, I get a DIVERGED_LINE_SEARCH. From
> what I have read, it seems this error mostly arises because of an incorrect
> Jacobian. But I think my Jacobian J looks correct, is it not?
>
> Or is this even the right approach?
>

I would just do simple FD (colocation) for this part of the problem. Then
you can see everything easily. Use
the FD Jacobian at first. You should converge if your guess is good enough.
Then put in an analytic Jacobian.

  Matt


> Thanks,
> Justin
>
>
> On Wed, Sep 9, 2015 at 4:41 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Sep 9, 2015 at 4:34 PM, Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> I need to solve this system of nonlinear geochemical reactions:
>>>
>>> psi_A = c_A + c_C
>>> psi_B = c_B + c_C
>>> c_C = k*c_A*c_B
>>>
>>> where psi_A and psi_B are sub components (i.e., a mixed system) of my
>>> mesh (this was done using firedrake) and k is a constant scalar. These
>>> quantities are known a prior, and psi_A/B was obtained from the
>>> advection-diffusion equation (using SUPG). Given the two-field formulation
>>> of psi_A/B, I need to obtain a three-field formulation of c_A/B/C using the
>>> above system of equations. It should be noted that the above system of
>>> equations are node-independent. That is, c_A/B/C of one node does not care
>>> what psi_A/B of other nodes may be.
>>>
>>> What?s the best strategy to go about solving this? With SciPy, i did
>>> something like this:
>>>
>>> #====================
>>> from scipy.optimize import fsolve
>>> import math
>>>
>>> # advection-diffusion for psi_A and psi_B
>>>
>>> # Nonlinear function for geochemical reactions
>>> def equations(p,psi_A,psi_B,k_1):
>>>  c_A,c_B,c_C = p
>>>  return (c_A+c_C-psi_A,c_B+c_C-psi_B,c_C-k_1*c_A*c_B)
>>>
>>> # Initialize
>>> c_A_vec = np.zeros(len(psi_A.vector()))
>>> c_B_vec = np.zeros(len(psi_A.vector()))
>>> c_C_vec = np.zeros(len(psi_A.vector()))
>>> k_1 = 1.0
>>>
>>> for i in range(len(psi_A.vector())):
>>>  c_A,c_B,c_C =
>>> fsolve(equations,(0.0,0.0,0.0),args=(psi_A.vector()[i],psi_B.vector()[i],k_1))
>>>  c_A_vec[i] = c_A
>>>  c_B_vec[i] = c_B
>>>  c_C=vec[i] = c_C
>>>
>>> c_A = Function(Q)
>>> c_B = Function(Q)
>>> c_C = Function(Q)
>>> c_A.vector()[:]=c_A_vec
>>> c_B.vector()[:]=c_B_vec
>>> c_C.vector()[:]=c_C_vec
>>> #======================
>>>
>>> The above is my temporary work-around to this issue. Basically, what I
>>> did was I solved the equations at each node. But is there a PETSc or
>>> petsc4py way to do this, specifically solving the above equations globally
>>> instead of at each individual node?
>>>
>>
>> 1) I had the same thing in DFT where I had to solve a local equation for
>> the screening length at each node
>>
>> 2) I would use a SNES
>>
>> 3) If you think there is variability in the solve, then do each
>> individually. Otherwise,
>>     you can easily do a Newton with a diagonal Jacobian.
>>
>> 4) I would eliminate c_C since its so easy
>>
>> 5) These look quadratic. Can't you solve this analytically?
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>>
>>> Thanks,
>>> Justin
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/47e6039e/attachment-0001.html>

From jychang48 at gmail.com  Fri Sep 11 06:56:31 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Fri, 11 Sep 2015 05:56:31 -0600
Subject: [petsc-users] Expressing this nonlinear system in PETSc and/or
	petsc4py
In-Reply-To: <CAMYG4Gk+U5xiT5HPt3+NkKAn4cRpKaND3xiS3DWfzwtt6N8png@mail.gmail.com>
References: <CAP2=TMiNLT7UK+MGGaRdmYwzvjEN4_h=hgO-5qMhjh-QOrtmYA@mail.gmail.com>
	<CAMYG4GkhhVyUSWzRnwWYAG93UuWPUR-q_1-=QsZXe15pS3oV+Q@mail.gmail.com>
	<CAP2=TMj86cQARWvSB6C9N-+57JtpTLvrnkiWbN2BRjaaxqUFTQ@mail.gmail.com>
	<CAMYG4Gk+U5xiT5HPt3+NkKAn4cRpKaND3xiS3DWfzwtt6N8png@mail.gmail.com>
Message-ID: <CAP2=TMjVwA7mypwyVjBUrrYsnspv6xKTE5WZnVH6kKBnvUELcw@mail.gmail.com>

Matt,

So it turns out that the issue I had was using c.split() as opposed to
split(c). The firedrake folks helped me resolve this (for now).

Thanks,
Justin

On Fri, Sep 11, 2015 at 5:45 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Sep 10, 2015 at 5:49 PM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> The example I have happens to be a simple system (e.g., Na^(+) + Cl^(-)
>> <=> NaCl), I could potentially solve a 7 component system with 4 primary
>> and 3 secondary species which in some cases cannot be solved analytically
>> as easily.
>>
>> Anyway, I was wondering if I could express the example system I had as a
>> mixed weak/variational form. That is, multiply each equation by a test
>> function and solve this system as if it were a finite element problem. I
>> had the following weak/variational form expressed in the FEniCS/Firedrake
>> form:
>>
>
> I do not see the advantage of a variational form here. There is no
> "energy" for the problem. Colocation seems easier.
>
>
>> #==========================
>> mesh = ...
>> Q = FunctionSpace(mesh, "CG", 1)
>> W = Q*Q*Q
>> dc_A,dc_B,dc_C = TrialFunctions(W)
>> q_A,q_B,q_C = TestFunctions(W)
>> c = Function(W)
>> c_A,c_B,c_C = c.split()
>>
>> ...
>>
>> F = (q_A*(u0_A - c_A - c_C) + q_B*(u0_B - c_B - c_C) + q_C*(c_C -
>> c_A*c_B))*dx
>> J = (-q_A*(dc_A + dc_C) + q_B*(dc_B + dc_C) + q_C*(dc_C - c_B*dc_A -
>> c_A*dc_B))*dx
>>
>> ...
>>
>> # Solve for u0_A a u0_B
>>
>> ....
>>
>> solve(F == 0, c, J=J)
>> #===========================
>>
>> Where u0_A and u0_B correspond to my psi_A/B from a different function
>> space. However, when I solve the system, I get a DIVERGED_LINE_SEARCH. From
>> what I have read, it seems this error mostly arises because of an incorrect
>> Jacobian. But I think my Jacobian J looks correct, is it not?
>>
>> Or is this even the right approach?
>>
>
> I would just do simple FD (colocation) for this part of the problem. Then
> you can see everything easily. Use
> the FD Jacobian at first. You should converge if your guess is good
> enough. Then put in an analytic Jacobian.
>
>   Matt
>
>
>> Thanks,
>> Justin
>>
>>
>> On Wed, Sep 9, 2015 at 4:41 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Wed, Sep 9, 2015 at 4:34 PM, Justin Chang <jychang48 at gmail.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I need to solve this system of nonlinear geochemical reactions:
>>>>
>>>> psi_A = c_A + c_C
>>>> psi_B = c_B + c_C
>>>> c_C = k*c_A*c_B
>>>>
>>>> where psi_A and psi_B are sub components (i.e., a mixed system) of my
>>>> mesh (this was done using firedrake) and k is a constant scalar. These
>>>> quantities are known a prior, and psi_A/B was obtained from the
>>>> advection-diffusion equation (using SUPG). Given the two-field formulation
>>>> of psi_A/B, I need to obtain a three-field formulation of c_A/B/C using the
>>>> above system of equations. It should be noted that the above system of
>>>> equations are node-independent. That is, c_A/B/C of one node does not care
>>>> what psi_A/B of other nodes may be.
>>>>
>>>> What?s the best strategy to go about solving this? With SciPy, i did
>>>> something like this:
>>>>
>>>> #====================
>>>> from scipy.optimize import fsolve
>>>> import math
>>>>
>>>> # advection-diffusion for psi_A and psi_B
>>>>
>>>> # Nonlinear function for geochemical reactions
>>>> def equations(p,psi_A,psi_B,k_1):
>>>>  c_A,c_B,c_C = p
>>>>  return (c_A+c_C-psi_A,c_B+c_C-psi_B,c_C-k_1*c_A*c_B)
>>>>
>>>> # Initialize
>>>> c_A_vec = np.zeros(len(psi_A.vector()))
>>>> c_B_vec = np.zeros(len(psi_A.vector()))
>>>> c_C_vec = np.zeros(len(psi_A.vector()))
>>>> k_1 = 1.0
>>>>
>>>> for i in range(len(psi_A.vector())):
>>>>  c_A,c_B,c_C =
>>>> fsolve(equations,(0.0,0.0,0.0),args=(psi_A.vector()[i],psi_B.vector()[i],k_1))
>>>>  c_A_vec[i] = c_A
>>>>  c_B_vec[i] = c_B
>>>>  c_C=vec[i] = c_C
>>>>
>>>> c_A = Function(Q)
>>>> c_B = Function(Q)
>>>> c_C = Function(Q)
>>>> c_A.vector()[:]=c_A_vec
>>>> c_B.vector()[:]=c_B_vec
>>>> c_C.vector()[:]=c_C_vec
>>>> #======================
>>>>
>>>> The above is my temporary work-around to this issue. Basically, what I
>>>> did was I solved the equations at each node. But is there a PETSc or
>>>> petsc4py way to do this, specifically solving the above equations globally
>>>> instead of at each individual node?
>>>>
>>>
>>> 1) I had the same thing in DFT where I had to solve a local equation for
>>> the screening length at each node
>>>
>>> 2) I would use a SNES
>>>
>>> 3) If you think there is variability in the solve, then do each
>>> individually. Otherwise,
>>>     you can easily do a Newton with a diagonal Jacobian.
>>>
>>> 4) I would eliminate c_C since its so easy
>>>
>>> 5) These look quadratic. Can't you solve this analytically?
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>>
>>>> Thanks,
>>>> Justin
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/d1f27aca/attachment.html>

From gideon.simpson at gmail.com  Fri Sep 11 08:15:54 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Fri, 11 Sep 2015 09:15:54 -0400
Subject: [petsc-users] grid sequencing & DMComposite
In-Reply-To: <69870B3A-E145-4902-A577-7D065025513E@mcs.anl.gov>
References: <68220B18-FFD0-473A-B87F-2C0FC1F279C3@gmail.com>
	<69870B3A-E145-4902-A577-7D065025513E@mcs.anl.gov>
Message-ID: <59842489-2781-4BD1-BE97-04319C23AFC2@gmail.com>

Ok, so here?s some behavior I don?t understand.  I am working with a DMComposite structure, and I do the following,

    SNESComputeFunction(snes,U_refine,r);
    SNESGetDM(snes, &dm_refine);

    DMCompositeGetLocalVectors(dm_refine, &rp, &rQ);
    DMCompositeScatter(dm_refine, r, rp, rQ);
      
    VecNorm(r, NORM_2, &r_norm);
    VecNorm(rp, NORM_2, &rp_norm);
    VecNorm(rQ, NORM_2, &rQ_norm);

    VecGetSize(r,&r_size);
    VecGetSize(rp, &rp_size);
    VecGetSize(rQ, &rQ_size);

    DMCompositeRestoreLocalVectors(dm_refine, &rp, &rQ);
    VecDestroy(&r);

    PetscPrintf(PETSC_COMM_WORLD," ||r|| = %g, %i entries\n",r_norm,r_size);
    PetscPrintf(PETSC_COMM_WORLD," ||rp|| = %g, %i entries\n",rp_norm,rp_size);
    PetscPrintf(PETSC_COMM_WORLD," ||rQ|| = %g, %i entries\n",rQ_norm,rQ_size);

and my output is:

 ||r|| = 225.31, 7999 entries
 ||rp|| = 140.021, 3 entries
 ||rQ|| = 176.56, 8004 entries


The arithmetic, is off, no?

-gideon

> On Sep 10, 2015, at 5:28 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  SNESGetDM()  this will return not the original DM you set but the refined one.
> 
>> On Sep 10, 2015, at 3:47 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> I?m using a DMCompsosite along with grid sequencing and here?s what I?m trying to accomplish.  After running the SNES solve, I?d like to evaluate the residual on the refined grid, on each piece of the DMComposite.  How do I get a DM for the refined grid which I can then use with DMCompositeGetLocalVectors in order to get each piece of the problem?  Or is there a better way?
>> 
>> -gideon
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/f365c03e/attachment.html>

From knepley at gmail.com  Fri Sep 11 08:36:01 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Sep 2015 08:36:01 -0500
Subject: [petsc-users] grid sequencing & DMComposite
In-Reply-To: <59842489-2781-4BD1-BE97-04319C23AFC2@gmail.com>
References: <68220B18-FFD0-473A-B87F-2C0FC1F279C3@gmail.com>
	<69870B3A-E145-4902-A577-7D065025513E@mcs.anl.gov>
	<59842489-2781-4BD1-BE97-04319C23AFC2@gmail.com>
Message-ID: <CAMYG4G=hnhW2qcbAzFTjmGXAJQgeAuixYQpdatia258vS9kN5Q@mail.gmail.com>

On Fri, Sep 11, 2015 at 8:15 AM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Ok, so here?s some behavior I don?t understand.  I am working with a
> DMComposite structure, and I do the following,
>
>     SNESComputeFunction(snes,U_refine,r);
>     SNESGetDM(snes, &dm_refine);
>
>     DMCompositeGetLocalVectors(dm_refine, &rp, &rQ);
>     DMCompositeScatter(dm_refine, r, rp, rQ);
>
>     VecNorm(r, NORM_2, &r_norm);
>     VecNorm(rp, NORM_2, &rp_norm);
>     VecNorm(rQ, NORM_2, &rQ_norm);
>
>     VecGetSize(r,&r_size);
>     VecGetSize(rp, &rp_size);
>     VecGetSize(rQ, &rQ_size);
>
>     DMCompositeRestoreLocalVectors(dm_refine, &rp, &rQ);
>     VecDestroy(&r);
>
>     PetscPrintf(PETSC_COMM_WORLD," ||r|| = %g, %i
> entries\n",r_norm,r_size);
>     PetscPrintf(PETSC_COMM_WORLD," ||rp|| = %g, %i
> entries\n",rp_norm,rp_size);
>     PetscPrintf(PETSC_COMM_WORLD," ||rQ|| = %g, %i
> entries\n",rQ_norm,rQ_size);
>
> and my output is:
>
>  ||r|| = 225.31, 7999 entries
>  ||rp|| = 140.021, 3 entries
>  ||rQ|| = 176.56, 8004 entries
>
>
> The arithmetic, is off, no?
>

I think you want Global vectors for this, not Local vectors.

  Matt


> -gideon
>
> On Sep 10, 2015, at 5:28 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>  SNESGetDM()  this will return not the original DM you set but the refined
> one.
>
> On Sep 10, 2015, at 3:47 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> I?m using a DMCompsosite along with grid sequencing and here?s what I?m
> trying to accomplish.  After running the SNES solve, I?d like to evaluate
> the residual on the refined grid, on each piece of the DMComposite.  How do
> I get a DM for the refined grid which I can then use with
> DMCompositeGetLocalVectors in order to get each piece of the problem?  Or
> is there a better way?
>
> -gideon
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/bc42082e/attachment-0001.html>

From gideon.simpson at gmail.com  Fri Sep 11 08:37:50 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Fri, 11 Sep 2015 09:37:50 -0400
Subject: [petsc-users] grid sequencing & DMComposite
In-Reply-To: <CAMYG4G=hnhW2qcbAzFTjmGXAJQgeAuixYQpdatia258vS9kN5Q@mail.gmail.com>
References: <68220B18-FFD0-473A-B87F-2C0FC1F279C3@gmail.com>
	<69870B3A-E145-4902-A577-7D065025513E@mcs.anl.gov>
	<59842489-2781-4BD1-BE97-04319C23AFC2@gmail.com>
	<CAMYG4G=hnhW2qcbAzFTjmGXAJQgeAuixYQpdatia258vS9kN5Q@mail.gmail.com>
Message-ID: <A84E785E-EF84-42C0-8F36-EDD3F1535B4E@gmail.com>

There doesn?t seem to be a global analog of DMCompositeGetLocalVectors.  Do I need to do manual indexing?

-gideon

> On Sep 11, 2015, at 9:36 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Sep 11, 2015 at 8:15 AM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> Ok, so here?s some behavior I don?t understand.  I am working with a DMComposite structure, and I do the following,
> 
>     SNESComputeFunction(snes,U_refine,r);
>     SNESGetDM(snes, &dm_refine);
> 
>     DMCompositeGetLocalVectors(dm_refine, &rp, &rQ);
>     DMCompositeScatter(dm_refine, r, rp, rQ);
>       
>     VecNorm(r, NORM_2, &r_norm);
>     VecNorm(rp, NORM_2, &rp_norm);
>     VecNorm(rQ, NORM_2, &rQ_norm);
> 
>     VecGetSize(r,&r_size);
>     VecGetSize(rp, &rp_size);
>     VecGetSize(rQ, &rQ_size);
> 
>     DMCompositeRestoreLocalVectors(dm_refine, &rp, &rQ);
>     VecDestroy(&r);
> 
>     PetscPrintf(PETSC_COMM_WORLD," ||r|| = %g, %i entries\n",r_norm,r_size);
>     PetscPrintf(PETSC_COMM_WORLD," ||rp|| = %g, %i entries\n",rp_norm,rp_size);
>     PetscPrintf(PETSC_COMM_WORLD," ||rQ|| = %g, %i entries\n",rQ_norm,rQ_size);
> 
> and my output is:
> 
>  ||r|| = 225.31, 7999 entries
>  ||rp|| = 140.021, 3 entries
>  ||rQ|| = 176.56, 8004 entries
> 
> 
> The arithmetic, is off, no?
> 
> I think you want Global vectors for this, not Local vectors.
> 
>   Matt
>  
> -gideon
> 
>> On Sep 10, 2015, at 5:28 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>> 
>> 
>>  SNESGetDM()  this will return not the original DM you set but the refined one.
>> 
>>> On Sep 10, 2015, at 3:47 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>> 
>>> I?m using a DMCompsosite along with grid sequencing and here?s what I?m trying to accomplish.  After running the SNES solve, I?d like to evaluate the residual on the refined grid, on each piece of the DMComposite.  How do I get a DM for the refined grid which I can then use with DMCompositeGetLocalVectors in order to get each piece of the problem?  Or is there a better way?
>>> 
>>> -gideon
>>> 
>> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/771c0a24/attachment.html>

From knepley at gmail.com  Fri Sep 11 08:47:21 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Sep 2015 08:47:21 -0500
Subject: [petsc-users] grid sequencing & DMComposite
In-Reply-To: <A84E785E-EF84-42C0-8F36-EDD3F1535B4E@gmail.com>
References: <68220B18-FFD0-473A-B87F-2C0FC1F279C3@gmail.com>
	<69870B3A-E145-4902-A577-7D065025513E@mcs.anl.gov>
	<59842489-2781-4BD1-BE97-04319C23AFC2@gmail.com>
	<CAMYG4G=hnhW2qcbAzFTjmGXAJQgeAuixYQpdatia258vS9kN5Q@mail.gmail.com>
	<A84E785E-EF84-42C0-8F36-EDD3F1535B4E@gmail.com>
Message-ID: <CAMYG4G=Qb9iBc4YYYAm+REnYsY+UuJxwRaxaq4L2B7ejHFqXsw@mail.gmail.com>

On Fri, Sep 11, 2015 at 8:37 AM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> There doesn?t seem to be a global analog of DMCompositeGetLocalVectors.
> Do I need to do manual indexing?
>

Its simple, lets just write it:


http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/composite/pack.c.html#DMCompositeGetLocalVectors

and from looking at the code here


http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/composite/pack.c.html#DMCompositeScatter

All we have to do is write one function which gives back a bunch of Global
vectors pointing at the same memory.
We use PlaceArray() on the Get call, and ResetArray() on the Restore call.

I can't believe this isn't in there already, but I think O(0) people use
DMComposite right now.

   Matt


> -gideon
>
> On Sep 11, 2015, at 9:36 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Fri, Sep 11, 2015 at 8:15 AM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
>> Ok, so here?s some behavior I don?t understand.  I am working with a
>> DMComposite structure, and I do the following,
>>
>>     SNESComputeFunction(snes,U_refine,r);
>>     SNESGetDM(snes, &dm_refine);
>>
>>     DMCompositeGetLocalVectors(dm_refine, &rp, &rQ);
>>     DMCompositeScatter(dm_refine, r, rp, rQ);
>>
>>     VecNorm(r, NORM_2, &r_norm);
>>     VecNorm(rp, NORM_2, &rp_norm);
>>     VecNorm(rQ, NORM_2, &rQ_norm);
>>
>>     VecGetSize(r,&r_size);
>>     VecGetSize(rp, &rp_size);
>>     VecGetSize(rQ, &rQ_size);
>>
>>     DMCompositeRestoreLocalVectors(dm_refine, &rp, &rQ);
>>     VecDestroy(&r);
>>
>>     PetscPrintf(PETSC_COMM_WORLD," ||r|| = %g, %i
>> entries\n",r_norm,r_size);
>>     PetscPrintf(PETSC_COMM_WORLD," ||rp|| = %g, %i
>> entries\n",rp_norm,rp_size);
>>     PetscPrintf(PETSC_COMM_WORLD," ||rQ|| = %g, %i
>> entries\n",rQ_norm,rQ_size);
>>
>> and my output is:
>>
>>  ||r|| = 225.31, 7999 entries
>>  ||rp|| = 140.021, 3 entries
>>  ||rQ|| = 176.56, 8004 entries
>>
>>
>> The arithmetic, is off, no?
>>
>
> I think you want Global vectors for this, not Local vectors.
>
>   Matt
>
>
>> -gideon
>>
>> On Sep 10, 2015, at 5:28 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>>  SNESGetDM()  this will return not the original DM you set but the
>> refined one.
>>
>> On Sep 10, 2015, at 3:47 PM, Gideon Simpson <gideon.simpson at gmail.com>
>> wrote:
>>
>> I?m using a DMCompsosite along with grid sequencing and here?s what I?m
>> trying to accomplish.  After running the SNES solve, I?d like to evaluate
>> the residual on the refined grid, on each piece of the DMComposite.  How do
>> I get a DM for the refined grid which I can then use with
>> DMCompositeGetLocalVectors in order to get each piece of the problem?  Or
>> is there a better way?
>>
>> -gideon
>>
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/322a7298/attachment.html>

From lawrence.mitchell at imperial.ac.uk  Fri Sep 11 08:48:39 2015
From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell)
Date: Fri, 11 Sep 2015 14:48:39 +0100
Subject: [petsc-users] grid sequencing & DMComposite
In-Reply-To: <A84E785E-EF84-42C0-8F36-EDD3F1535B4E@gmail.com>
References: <68220B18-FFD0-473A-B87F-2C0FC1F279C3@gmail.com>
	<69870B3A-E145-4902-A577-7D065025513E@mcs.anl.gov>
	<59842489-2781-4BD1-BE97-04319C23AFC2@gmail.com>
	<CAMYG4G=hnhW2qcbAzFTjmGXAJQgeAuixYQpdatia258vS9kN5Q@mail.gmail.com>
	<A84E785E-EF84-42C0-8F36-EDD3F1535B4E@gmail.com>
Message-ID: <D318A995-DE6A-4030-96B0-068630F4CA43@imperial.ac.uk>


> On 11 Sep 2015, at 14:37, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> There doesn?t seem to be a global analog of DMCompositeGetLocalVectors.  Do I need to do manual indexing?

I think you use DMCompositeGetAccess.

Lawrence
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/15ef3a9c/attachment.pgp>

From knepley at gmail.com  Fri Sep 11 08:53:20 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Sep 2015 08:53:20 -0500
Subject: [petsc-users] grid sequencing & DMComposite
In-Reply-To: <D318A995-DE6A-4030-96B0-068630F4CA43@imperial.ac.uk>
References: <68220B18-FFD0-473A-B87F-2C0FC1F279C3@gmail.com>
	<69870B3A-E145-4902-A577-7D065025513E@mcs.anl.gov>
	<59842489-2781-4BD1-BE97-04319C23AFC2@gmail.com>
	<CAMYG4G=hnhW2qcbAzFTjmGXAJQgeAuixYQpdatia258vS9kN5Q@mail.gmail.com>
	<A84E785E-EF84-42C0-8F36-EDD3F1535B4E@gmail.com>
	<D318A995-DE6A-4030-96B0-068630F4CA43@imperial.ac.uk>
Message-ID: <CAMYG4G=85EKkhtEkYhXGr6acCuNqq+OTm4CpxcdfK=sRrgkv8g@mail.gmail.com>

On Fri, Sep 11, 2015 at 8:48 AM, Lawrence Mitchell <
lawrence.mitchell at imperial.ac.uk> wrote:

>
> > On 11 Sep 2015, at 14:37, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> >
> > There doesn?t seem to be a global analog of DMCompositeGetLocalVectors.
> Do I need to do manual indexing?
>
> I think you use DMCompositeGetAccess.


Okay, this is just flippin' crazy. We have DMCompositeGetLocalVectors(),
but DMCompositeGetAccess()???

This needs to be changed to DMCompositeGetGlobalVectors().

   Matt


>
> Lawrence
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/34ea26f3/attachment.html>

From gideon.simpson at gmail.com  Fri Sep 11 08:57:40 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Fri, 11 Sep 2015 09:57:40 -0400
Subject: [petsc-users] grid sequencing & DMComposite
In-Reply-To: <D318A995-DE6A-4030-96B0-068630F4CA43@imperial.ac.uk>
References: <68220B18-FFD0-473A-B87F-2C0FC1F279C3@gmail.com>
	<69870B3A-E145-4902-A577-7D065025513E@mcs.anl.gov>
	<59842489-2781-4BD1-BE97-04319C23AFC2@gmail.com>
	<CAMYG4G=hnhW2qcbAzFTjmGXAJQgeAuixYQpdatia258vS9kN5Q@mail.gmail.com>
	<A84E785E-EF84-42C0-8F36-EDD3F1535B4E@gmail.com>
	<D318A995-DE6A-4030-96B0-068630F4CA43@imperial.ac.uk>
Message-ID: <0BAAE356-650F-48F2-8A35-1443AD79342E@gmail.com>

Yea, that works as advertised.

-gideon

> On Sep 11, 2015, at 9:48 AM, Lawrence Mitchell <lawrence.mitchell at imperial.ac.uk> wrote:
> 
> 
>> On 11 Sep 2015, at 14:37, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> There doesn?t seem to be a global analog of DMCompositeGetLocalVectors.  Do I need to do manual indexing?
> 
> I think you use DMCompositeGetAccess.
> 
> Lawrence

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/27c4645b/attachment-0001.html>

From gideon.simpson at gmail.com  Fri Sep 11 09:08:08 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Fri, 11 Sep 2015 10:08:08 -0400
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
Message-ID: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>

Following up on the previous thread, for my dm composite problem, I find that at the end of the first grid sequence,where it fails to converge, the distribution of the norms between the two pieces are:

 39 SNES Function norm 2.253098577796e+02 
 40 SNES Function norm 2.253098577331e+02 
 41 SNES Function norm 2.253098577228e+02 
 42 SNES Function norm 2.253098577212e+02 
 43 SNES Function norm 2.253098577174e+02 
 44 SNES Function norm 2.253098577166e+02 
 45 SNES Function norm 2.253098577158e+02 
 46 SNES Function norm 2.253098577157e+02 
 47 SNES Function norm 2.253098577156e+02 
 48 SNES Function norm 2.253098577156e+02 
Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 48
 ||r|| = 225.31, 7999 entries
 ||rp|| = 140.021, 3 entries
 ||rQ|| = 176.518, 7996 entries

Since I think we were convinced that this was intrinsic to the problem, and not a function of the Jacobian function, I am using my Jacobian.

-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/6a8a837b/attachment.html>

From knepley at gmail.com  Fri Sep 11 09:16:12 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Sep 2015 09:16:12 -0500
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
Message-ID: <CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>

On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Following up on the previous thread, for my dm composite problem, I find
> that at the end of the first grid sequence,where it fails to converge, the
> distribution of the norms between the two pieces are:
>
>  39 SNES Function norm 2.253098577796e+02
>  40 SNES Function norm 2.253098577331e+02
>  41 SNES Function norm 2.253098577228e+02
>  42 SNES Function norm 2.253098577212e+02
>  43 SNES Function norm 2.253098577174e+02
>  44 SNES Function norm 2.253098577166e+02
>  45 SNES Function norm 2.253098577158e+02
>  46 SNES Function norm 2.253098577157e+02
>  47 SNES Function norm 2.253098577156e+02
>  48 SNES Function norm 2.253098577156e+02
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 48
>  ||r|| = 225.31, 7999 entries
>  ||rp|| = 140.021, 3 entries
>  ||rQ|| = 176.518, 7996 entries
>
> Since I think we were convinced that this was intrinsic to the problem,
> and not a function of the Jacobian function, I am using my Jacobian.
>

Okay, I see no pattern in the fields. Lets plot these 2 vectors, -vec_view
draw, and screenshot.

  Matt


> -gideon
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/20a6e309/attachment.html>

From gideon.simpson at gmail.com  Fri Sep 11 13:05:41 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Fri, 11 Sep 2015 14:05:41 -0400
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
Message-ID: <D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>

Since the problem has not only the two components in the DM, but the second component has 4 degrees of freedom per mesh point, I thought it best to do the post processing separately.  See attached

-gideon

> On Sep 11, 2015, at 10:16 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> Following up on the previous thread, for my dm composite problem, I find that at the end of the first grid sequence,where it fails to converge, the distribution of the norms between the two pieces are:
> 
>  39 SNES Function norm 2.253098577796e+02 
>  40 SNES Function norm 2.253098577331e+02 
>  41 SNES Function norm 2.253098577228e+02 
>  42 SNES Function norm 2.253098577212e+02 
>  43 SNES Function norm 2.253098577174e+02 
>  44 SNES Function norm 2.253098577166e+02 
>  45 SNES Function norm 2.253098577158e+02 
>  46 SNES Function norm 2.253098577157e+02 
>  47 SNES Function norm 2.253098577156e+02 
>  48 SNES Function norm 2.253098577156e+02 
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 48
>  ||r|| = 225.31, 7999 entries
>  ||rp|| = 140.021, 3 entries
>  ||rQ|| = 176.518, 7996 entries
> 
> Since I think we were convinced that this was intrinsic to the problem, and not a function of the Jacobian function, I am using my Jacobian.
> 
> Okay, I see no pattern in the fields. Lets plot these 2 vectors, -vec_view draw, and screenshot.
> 
>   Matt
>  
> -gideon
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/efac7f14/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2015-09-11 at 2.04.25 PM.png
Type: image/png
Size: 261398 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/efac7f14/attachment-0001.png>

From knepley at gmail.com  Fri Sep 11 15:05:38 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Sep 2015 15:05:38 -0500
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
Message-ID: <CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>

On Fri, Sep 11, 2015 at 1:05 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Since the problem has not only the two components in the DM, but the
> second component has 4 degrees of freedom per mesh point, I thought it best
> to do the post processing separately.  See attached
>

So the whole thing is being controlled by 1 variable.

We should try freezing everything else, and just solving that scalar
equation I guess.

   Matt


> -gideon
>
> On Sep 11, 2015, at 10:16 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <gideon.simpson at gmail.com>
>  wrote:
>
>> Following up on the previous thread, for my dm composite problem, I find
>> that at the end of the first grid sequence,where it fails to converge, the
>> distribution of the norms between the two pieces are:
>>
>>  39 SNES Function norm 2.253098577796e+02
>>  40 SNES Function norm 2.253098577331e+02
>>  41 SNES Function norm 2.253098577228e+02
>>  42 SNES Function norm 2.253098577212e+02
>>  43 SNES Function norm 2.253098577174e+02
>>  44 SNES Function norm 2.253098577166e+02
>>  45 SNES Function norm 2.253098577158e+02
>>  46 SNES Function norm 2.253098577157e+02
>>  47 SNES Function norm 2.253098577156e+02
>>  48 SNES Function norm 2.253098577156e+02
>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 48
>>  ||r|| = 225.31, 7999 entries
>>  ||rp|| = 140.021, 3 entries
>>  ||rQ|| = 176.518, 7996 entries
>>
>> Since I think we were convinced that this was intrinsic to the problem,
>> and not a function of the Jacobian function, I am using my Jacobian.
>>
>
> Okay, I see no pattern in the fields. Lets plot these 2 vectors, -vec_view
> draw, and screenshot.
>
>   Matt
>
>
>> -gideon
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/4e82f9a1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2015-09-11 at 2.04.25 PM.png
Type: image/png
Size: 261398 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/4e82f9a1/attachment-0001.png>

From gideon.simpson at gmail.com  Fri Sep 11 16:30:22 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Fri, 11 Sep 2015 17:30:22 -0400
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
Message-ID: <5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>

Are there any built in routines for freezing variables in SNES, or will that need to be handled by hand.

Also, I remain curious about the starting guess that the grid sequence uses during each refinement.  Is there a way to dump those to disk for inspection?

-gideon

> On Sep 11, 2015, at 4:05 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Sep 11, 2015 at 1:05 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> Since the problem has not only the two components in the DM, but the second component has 4 degrees of freedom per mesh point, I thought it best to do the post processing separately.  See attached
> 
> So the whole thing is being controlled by 1 variable.
> 
> We should try freezing everything else, and just solving that scalar equation I guess.
> 
>    Matt
>  
> -gideon
> <Screen Shot 2015-09-11 at 2.04.25 PM.png>
>> On Sep 11, 2015, at 10:16 AM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>> 
>> On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>> Following up on the previous thread, for my dm composite problem, I find that at the end of the first grid sequence,where it fails to converge, the distribution of the norms between the two pieces are:
>> 
>>  39 SNES Function norm 2.253098577796e+02 
>>  40 SNES Function norm 2.253098577331e+02 
>>  41 SNES Function norm 2.253098577228e+02 
>>  42 SNES Function norm 2.253098577212e+02 
>>  43 SNES Function norm 2.253098577174e+02 
>>  44 SNES Function norm 2.253098577166e+02 
>>  45 SNES Function norm 2.253098577158e+02 
>>  46 SNES Function norm 2.253098577157e+02 
>>  47 SNES Function norm 2.253098577156e+02 
>>  48 SNES Function norm 2.253098577156e+02 
>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 48
>>  ||r|| = 225.31, 7999 entries
>>  ||rp|| = 140.021, 3 entries
>>  ||rQ|| = 176.518, 7996 entries
>> 
>> Since I think we were convinced that this was intrinsic to the problem, and not a function of the Jacobian function, I am using my Jacobian.
>> 
>> Okay, I see no pattern in the fields. Lets plot these 2 vectors, -vec_view draw, and screenshot.
>> 
>>   Matt
>>  
>> -gideon
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150911/7b3c5fb2/attachment.html>

From bsmith at mcs.anl.gov  Fri Sep 11 22:48:29 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 11 Sep 2015 22:48:29 -0500
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
	<5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
Message-ID: <9C639A70-1082-41FF-A746-631594BF8772@mcs.anl.gov>


> On Sep 11, 2015, at 4:30 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> Are there any built in routines for freezing variables in SNES, or will that need to be handled by hand.
> 
> Also, I remain curious about the starting guess that the grid sequence uses during each refinement.  Is there a way to dump those to disk for inspection?
  
  Not really. You can look in SNESSolve() and see where the interpolation takes place and add your own call to VecView() to save the result.

  Barry

> 
> -gideon
> 
>> On Sep 11, 2015, at 4:05 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Fri, Sep 11, 2015 at 1:05 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> Since the problem has not only the two components in the DM, but the second component has 4 degrees of freedom per mesh point, I thought it best to do the post processing separately.  See attached
>> 
>> So the whole thing is being controlled by 1 variable.
>> 
>> We should try freezing everything else, and just solving that scalar equation I guess.
>> 
>>    Matt
>>  
>> -gideon
>> <Screen Shot 2015-09-11 at 2.04.25 PM.png>
>>> On Sep 11, 2015, at 10:16 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>> On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> Following up on the previous thread, for my dm composite problem, I find that at the end of the first grid sequence,where it fails to converge, the distribution of the norms between the two pieces are:
>>> 
>>>  39 SNES Function norm 2.253098577796e+02 
>>>  40 SNES Function norm 2.253098577331e+02 
>>>  41 SNES Function norm 2.253098577228e+02 
>>>  42 SNES Function norm 2.253098577212e+02 
>>>  43 SNES Function norm 2.253098577174e+02 
>>>  44 SNES Function norm 2.253098577166e+02 
>>>  45 SNES Function norm 2.253098577158e+02 
>>>  46 SNES Function norm 2.253098577157e+02 
>>>  47 SNES Function norm 2.253098577156e+02 
>>>  48 SNES Function norm 2.253098577156e+02 
>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 48
>>>  ||r|| = 225.31, 7999 entries
>>>  ||rp|| = 140.021, 3 entries
>>>  ||rQ|| = 176.518, 7996 entries
>>> 
>>> Since I think we were convinced that this was intrinsic to the problem, and not a function of the Jacobian function, I am using my Jacobian.
>>> 
>>> Okay, I see no pattern in the fields. Lets plot these 2 vectors, -vec_view draw, and screenshot.
>>> 
>>>   Matt
>>>  
>>> -gideon
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 


From knepley at gmail.com  Sat Sep 12 06:14:47 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 12 Sep 2015 06:14:47 -0500
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
	<5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
Message-ID: <CAMYG4G=uZxLZG8drNnfgKi7M98y+=FQXRAsdsZBbHwpcHFhRWw@mail.gmail.com>

On Fri, Sep 11, 2015 at 4:30 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Are there any built in routines for freezing variables in SNES, or will
> that need to be handled by hand.
>

We used to have this for KSP, but it looks like someone removed it. The
only thing left is KSPMonitorRange().
What we did is find the few largest residual elements, take a small halo
around them, project the problem to this
small space using MatGetSubMatrix() and VecScatter, solve, and VecScatter
back.

We do not have this for nonlinear stuff (like many other things) because
there is no explicit matrix to manipulate,
and language support for computing only parts of the nonlinear function is
really weak. What we really want
is something like what Victor Eijkout was proposing a few years ago, namely
automatic discovery of index sets
for communication, in this case with main memory. We would need the
residual code, given an output set, to tell
us what input set is needed. Then we make a Scatter, select part of the DM,
and we could compute. Now it has
to be done by hand.

  Thanks,

     Matt


> Also, I remain curious about the starting guess that the grid sequence
> uses during each refinement.  Is there a way to dump those to disk for
> inspection?
>
> -gideon
>
> On Sep 11, 2015, at 4:05 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Fri, Sep 11, 2015 at 1:05 PM, Gideon Simpson <gideon.simpson at gmail.com>
>  wrote:
>
>> Since the problem has not only the two components in the DM, but the
>> second component has 4 degrees of freedom per mesh point, I thought it best
>> to do the post processing separately.  See attached
>>
>
> So the whole thing is being controlled by 1 variable.
>
> We should try freezing everything else, and just solving that scalar
> equation I guess.
>
>    Matt
>
>
>> -gideon
>> <Screen Shot 2015-09-11 at 2.04.25 PM.png>
>>
>> On Sep 11, 2015, at 10:16 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <gideon.simpson at gmail.com
>> > wrote:
>>
>>> Following up on the previous thread, for my dm composite problem, I find
>>> that at the end of the first grid sequence,where it fails to converge, the
>>> distribution of the norms between the two pieces are:
>>>
>>>  39 SNES Function norm 2.253098577796e+02
>>>  40 SNES Function norm 2.253098577331e+02
>>>  41 SNES Function norm 2.253098577228e+02
>>>  42 SNES Function norm 2.253098577212e+02
>>>  43 SNES Function norm 2.253098577174e+02
>>>  44 SNES Function norm 2.253098577166e+02
>>>  45 SNES Function norm 2.253098577158e+02
>>>  46 SNES Function norm 2.253098577157e+02
>>>  47 SNES Function norm 2.253098577156e+02
>>>  48 SNES Function norm 2.253098577156e+02
>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations
>>> 48
>>>  ||r|| = 225.31, 7999 entries
>>>  ||rp|| = 140.021, 3 entries
>>>  ||rQ|| = 176.518, 7996 entries
>>>
>>> Since I think we were convinced that this was intrinsic to the problem,
>>> and not a function of the Jacobian function, I am using my Jacobian.
>>>
>>
>> Okay, I see no pattern in the fields. Lets plot these 2 vectors,
>> -vec_view draw, and screenshot.
>>
>>   Matt
>>
>>
>>> -gideon
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150912/26b52740/attachment.html>

From gideon.simpson at gmail.com  Sat Sep 12 13:26:39 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Sat, 12 Sep 2015 14:26:39 -0400
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <CAMYG4G=uZxLZG8drNnfgKi7M98y+=FQXRAsdsZBbHwpcHFhRWw@mail.gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
	<5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
	<CAMYG4G=uZxLZG8drNnfgKi7M98y+=FQXRAsdsZBbHwpcHFhRWw@mail.gmail.com>
Message-ID: <A502B698-F0B6-4084-95A2-4D694DCF3CAB@gmail.com>

Got it.  I wanted to follow up on something that I had been suspicious of for some time.  I took my initial solution, the one which we want to apply grid sequencing to, and did the following.  I loved into spicy, and manually did grid refinement using SciPy, dumped to disk, and then ran that through petsc, with the refined mesh.  When I say ?manual? here, I just mean that I called the SciPy spline interplant commands on uniform meshes that had twice the resolution, not that I manually picked out the refinement points.  When I use a linear interpolant, and run my solver on it, it fails miserably, just as the first grid sequence refinement does.  The numbers are a little different, but the residuals are still O(100).

If, instead, I use a second order interpolant, it solves it 11 iterations, wiithout a problem, and the result is physically consistent.  While this may be two sides of the same coin, it makes me think that the real challenge in my is the interpolation strategy used under grid refinement.  Is there a way to, manually, I suppose, set the interpolation that will be used under the grid sequence?



-gideon

> On Sep 12, 2015, at 7:14 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Sep 11, 2015 at 4:30 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> Are there any built in routines for freezing variables in SNES, or will that need to be handled by hand.
> 
> We used to have this for KSP, but it looks like someone removed it. The only thing left is KSPMonitorRange().
> What we did is find the few largest residual elements, take a small halo around them, project the problem to this
> small space using MatGetSubMatrix() and VecScatter, solve, and VecScatter back.
> 
> We do not have this for nonlinear stuff (like many other things) because there is no explicit matrix to manipulate,
> and language support for computing only parts of the nonlinear function is really weak. What we really want
> is something like what Victor Eijkout was proposing a few years ago, namely automatic discovery of index sets
> for communication, in this case with main memory. We would need the residual code, given an output set, to tell
> us what input set is needed. Then we make a Scatter, select part of the DM, and we could compute. Now it has
> to be done by hand.
> 
>   Thanks,
> 
>      Matt
>  
> Also, I remain curious about the starting guess that the grid sequence uses during each refinement.  Is there a way to dump those to disk for inspection?
> 
> -gideon
> 
>> On Sep 11, 2015, at 4:05 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>> 
>> On Fri, Sep 11, 2015 at 1:05 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>> Since the problem has not only the two components in the DM, but the second component has 4 degrees of freedom per mesh point, I thought it best to do the post processing separately.  See attached
>> 
>> So the whole thing is being controlled by 1 variable.
>> 
>> We should try freezing everything else, and just solving that scalar equation I guess.
>> 
>>    Matt
>>  
>> -gideon
>> <Screen Shot 2015-09-11 at 2.04.25 PM.png>
>> 
>>> On Sep 11, 2015, at 10:16 AM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>> 
>>> On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>> Following up on the previous thread, for my dm composite problem, I find that at the end of the first grid sequence,where it fails to converge, the distribution of the norms between the two pieces are:
>>> 
>>>  39 SNES Function norm 2.253098577796e+02 
>>>  40 SNES Function norm 2.253098577331e+02 
>>>  41 SNES Function norm 2.253098577228e+02 
>>>  42 SNES Function norm 2.253098577212e+02 
>>>  43 SNES Function norm 2.253098577174e+02 
>>>  44 SNES Function norm 2.253098577166e+02 
>>>  45 SNES Function norm 2.253098577158e+02 
>>>  46 SNES Function norm 2.253098577157e+02 
>>>  47 SNES Function norm 2.253098577156e+02 
>>>  48 SNES Function norm 2.253098577156e+02 
>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 48
>>>  ||r|| = 225.31, 7999 entries
>>>  ||rp|| = 140.021, 3 entries
>>>  ||rQ|| = 176.518, 7996 entries
>>> 
>>> Since I think we were convinced that this was intrinsic to the problem, and not a function of the Jacobian function, I am using my Jacobian.
>>> 
>>> Okay, I see no pattern in the fields. Lets plot these 2 vectors, -vec_view draw, and screenshot.
>>> 
>>>   Matt
>>>  
>>> -gideon
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150912/663e276e/attachment.html>

From knepley at gmail.com  Sat Sep 12 13:36:57 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 12 Sep 2015 13:36:57 -0500
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <A502B698-F0B6-4084-95A2-4D694DCF3CAB@gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
	<5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
	<CAMYG4G=uZxLZG8drNnfgKi7M98y+=FQXRAsdsZBbHwpcHFhRWw@mail.gmail.com>
	<A502B698-F0B6-4084-95A2-4D694DCF3CAB@gmail.com>
Message-ID: <CAMYG4Gn6HdUim1Y1FY5=W4HCjO=LmVCYknm+xxnk_s=YXLbcMg@mail.gmail.com>

On Sat, Sep 12, 2015 at 1:26 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Got it.  I wanted to follow up on something that I had been suspicious of
> for some time.  I took my initial solution, the one which we want to apply
> grid sequencing to, and did the following.  I loved into spicy, and
> manually did grid refinement using SciPy, dumped to disk, and then ran that
> through petsc, with the refined mesh.  When I say ?manual? here, I just
> mean that I called the SciPy spline interplant commands on uniform meshes
> that had twice the resolution, not that I manually picked out the
> refinement points.  When I use a linear interpolant, and run my solver on
> it, it fails miserably, just as the first grid sequence refinement does.
> The numbers are a little different, but the residuals are still O(100).
>
> If, instead, I use a second order interpolant, it solves it 11 iterations,
> wiithout a problem, and the result is physically consistent.  While this
> may be two sides of the same coin, it makes me think that the real
> challenge in my is the interpolation strategy used under grid refinement.
> Is there a way to, manually, I suppose, set the interpolation that will be
> used under the grid sequence?
>

Excellent! And also somewhat bizarre. This would be great to analyze. Is
there any bribe we could give you to
convert this to a real PETSc example? It would be a nice way to show that
dumping in high frequency energy
when interpolating can be bad for coupled problems.

Actual Solution: I think this will involve some coding. The quick and dirty
way is to just stick the code you want
into dainterp.c:DMCreateInterpolation_DA(). If that works, we add it to
interptype and enable it from the command
line. It would be nice to have a spline interpolant.

  Thanks,

     Matt


> -gideon
>
> On Sep 12, 2015, at 7:14 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Fri, Sep 11, 2015 at 4:30 PM, Gideon Simpson <gideon.simpson at gmail.com>
>  wrote:
>
>> Are there any built in routines for freezing variables in SNES, or will
>> that need to be handled by hand.
>>
>
> We used to have this for KSP, but it looks like someone removed it. The
> only thing left is KSPMonitorRange().
> What we did is find the few largest residual elements, take a small halo
> around them, project the problem to this
> small space using MatGetSubMatrix() and VecScatter, solve, and VecScatter
> back.
>
> We do not have this for nonlinear stuff (like many other things) because
> there is no explicit matrix to manipulate,
> and language support for computing only parts of the nonlinear function is
> really weak. What we really want
> is something like what Victor Eijkout was proposing a few years ago,
> namely automatic discovery of index sets
> for communication, in this case with main memory. We would need the
> residual code, given an output set, to tell
> us what input set is needed. Then we make a Scatter, select part of the
> DM, and we could compute. Now it has
> to be done by hand.
>
>   Thanks,
>
>      Matt
>
>
>> Also, I remain curious about the starting guess that the grid sequence
>> uses during each refinement.  Is there a way to dump those to disk for
>> inspection?
>>
>> -gideon
>>
>> On Sep 11, 2015, at 4:05 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Fri, Sep 11, 2015 at 1:05 PM, Gideon Simpson <gideon.simpson at gmail.com
>> > wrote:
>>
>>> Since the problem has not only the two components in the DM, but the
>>> second component has 4 degrees of freedom per mesh point, I thought it best
>>> to do the post processing separately.  See attached
>>>
>>
>> So the whole thing is being controlled by 1 variable.
>>
>> We should try freezing everything else, and just solving that scalar
>> equation I guess.
>>
>>    Matt
>>
>>
>>> -gideon
>>> <Screen Shot 2015-09-11 at 2.04.25 PM.png>
>>>
>>> On Sep 11, 2015, at 10:16 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>> On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <
>>> gideon.simpson at gmail.com> wrote:
>>>
>>>> Following up on the previous thread, for my dm composite problem, I
>>>> find that at the end of the first grid sequence,where it fails to converge,
>>>> the distribution of the norms between the two pieces are:
>>>>
>>>>  39 SNES Function norm 2.253098577796e+02
>>>>  40 SNES Function norm 2.253098577331e+02
>>>>  41 SNES Function norm 2.253098577228e+02
>>>>  42 SNES Function norm 2.253098577212e+02
>>>>  43 SNES Function norm 2.253098577174e+02
>>>>  44 SNES Function norm 2.253098577166e+02
>>>>  45 SNES Function norm 2.253098577158e+02
>>>>  46 SNES Function norm 2.253098577157e+02
>>>>  47 SNES Function norm 2.253098577156e+02
>>>>  48 SNES Function norm 2.253098577156e+02
>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations
>>>> 48
>>>>  ||r|| = 225.31, 7999 entries
>>>>  ||rp|| = 140.021, 3 entries
>>>>  ||rQ|| = 176.518, 7996 entries
>>>>
>>>> Since I think we were convinced that this was intrinsic to the problem,
>>>> and not a function of the Jacobian function, I am using my Jacobian.
>>>>
>>>
>>> Okay, I see no pattern in the fields. Lets plot these 2 vectors,
>>> -vec_view draw, and screenshot.
>>>
>>>   Matt
>>>
>>>
>>>> -gideon
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150912/122cf84d/attachment-0001.html>

From gideon.simpson at gmail.com  Sat Sep 12 13:54:31 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Sat, 12 Sep 2015 14:54:31 -0400
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <CAMYG4Gn6HdUim1Y1FY5=W4HCjO=LmVCYknm+xxnk_s=YXLbcMg@mail.gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
	<5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
	<CAMYG4G=uZxLZG8drNnfgKi7M98y+=FQXRAsdsZBbHwpcHFhRWw@mail.gmail.com>
	<A502B698-F0B6-4084-95A2-4D694DCF3CAB@gmail.com>
	<CAMYG4Gn6HdUim1Y1FY5=W4HCjO=LmVCYknm+xxnk_s=YXLbcMg@mail.gmail.com>
Message-ID: <B81E8617-6EBB-45C2-8C3D-F7EC26E51215@gmail.com>

I?m not sure how good a problem this would be as a toy model, but, once this is all done, I?d be happy to share the code.  

I?ve got a few comments on this:

1.  The underlying problem is in the study of derivative nonlinear Schr?dinger equations - see http://arxiv.org/abs/1301.1048 <http://arxiv.org/abs/1301.1048> for details of the equation.  The problem has features of the Airy equation, which as you?d guess, has exactly this problem of highly oscillatory structures which don?t get along with linear interpolation.

2.  The problem is complex valued with a real and imaginary components at each site on the mesh.  At the origin, the boundary conditions are Dirichlet on the imaginary component, and Neumann on the real component.  This led me to make the value of the real component at the origin, u0, one of the three parameters in the first component of the DMComposite structure.  Consequently, (u0, 0), the values of the function at the origin, should be involved in the interpolation strategy.  Is there a way, through ghost points, for the current DM interpolation strategy to make use of these under refinement?  Does it use ghost point values?

3.  Is DMShellSetCreateInterpolation what I want to use here?  Where I would define my own interpolation strategy across dm?s?  

-gideon

> On Sep 12, 2015, at 2:36 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Sat, Sep 12, 2015 at 1:26 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> Got it.  I wanted to follow up on something that I had been suspicious of for some time.  I took my initial solution, the one which we want to apply grid sequencing to, and did the following.  I loved into spicy, and manually did grid refinement using SciPy, dumped to disk, and then ran that through petsc, with the refined mesh.  When I say ?manual? here, I just mean that I called the SciPy spline interplant commands on uniform meshes that had twice the resolution, not that I manually picked out the refinement points.  When I use a linear interpolant, and run my solver on it, it fails miserably, just as the first grid sequence refinement does.  The numbers are a little different, but the residuals are still O(100).
> 
> If, instead, I use a second order interpolant, it solves it 11 iterations, wiithout a problem, and the result is physically consistent.  While this may be two sides of the same coin, it makes me think that the real challenge in my is the interpolation strategy used under grid refinement.  Is there a way to, manually, I suppose, set the interpolation that will be used under the grid sequence?
> 
> Excellent! And also somewhat bizarre. This would be great to analyze. Is there any bribe we could give you to
> convert this to a real PETSc example? It would be a nice way to show that dumping in high frequency energy
> when interpolating can be bad for coupled problems.
> 
> Actual Solution: I think this will involve some coding. The quick and dirty way is to just stick the code you want
> into dainterp.c:DMCreateInterpolation_DA(). If that works, we add it to interptype and enable it from the command
> line. It would be nice to have a spline interpolant.
> 
>   Thanks,
> 
>      Matt
>  
> -gideon
> 
>> On Sep 12, 2015, at 7:14 AM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>> 
>> On Fri, Sep 11, 2015 at 4:30 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>> Are there any built in routines for freezing variables in SNES, or will that need to be handled by hand.
>> 
>> We used to have this for KSP, but it looks like someone removed it. The only thing left is KSPMonitorRange().
>> What we did is find the few largest residual elements, take a small halo around them, project the problem to this
>> small space using MatGetSubMatrix() and VecScatter, solve, and VecScatter back.
>> 
>> We do not have this for nonlinear stuff (like many other things) because there is no explicit matrix to manipulate,
>> and language support for computing only parts of the nonlinear function is really weak. What we really want
>> is something like what Victor Eijkout was proposing a few years ago, namely automatic discovery of index sets
>> for communication, in this case with main memory. We would need the residual code, given an output set, to tell
>> us what input set is needed. Then we make a Scatter, select part of the DM, and we could compute. Now it has
>> to be done by hand.
>> 
>>   Thanks,
>> 
>>      Matt
>>  
>> Also, I remain curious about the starting guess that the grid sequence uses during each refinement.  Is there a way to dump those to disk for inspection?
>> 
>> -gideon
>> 
>>> On Sep 11, 2015, at 4:05 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>> 
>>> On Fri, Sep 11, 2015 at 1:05 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>> Since the problem has not only the two components in the DM, but the second component has 4 degrees of freedom per mesh point, I thought it best to do the post processing separately.  See attached
>>> 
>>> So the whole thing is being controlled by 1 variable.
>>> 
>>> We should try freezing everything else, and just solving that scalar equation I guess.
>>> 
>>>    Matt
>>>  
>>> -gideon
>>> <Screen Shot 2015-09-11 at 2.04.25 PM.png>
>>> 
>>>> On Sep 11, 2015, at 10:16 AM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>> 
>>>> On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
>>>> Following up on the previous thread, for my dm composite problem, I find that at the end of the first grid sequence,where it fails to converge, the distribution of the norms between the two pieces are:
>>>> 
>>>>  39 SNES Function norm 2.253098577796e+02 
>>>>  40 SNES Function norm 2.253098577331e+02 
>>>>  41 SNES Function norm 2.253098577228e+02 
>>>>  42 SNES Function norm 2.253098577212e+02 
>>>>  43 SNES Function norm 2.253098577174e+02 
>>>>  44 SNES Function norm 2.253098577166e+02 
>>>>  45 SNES Function norm 2.253098577158e+02 
>>>>  46 SNES Function norm 2.253098577157e+02 
>>>>  47 SNES Function norm 2.253098577156e+02 
>>>>  48 SNES Function norm 2.253098577156e+02 
>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 48
>>>>  ||r|| = 225.31, 7999 entries
>>>>  ||rp|| = 140.021, 3 entries
>>>>  ||rQ|| = 176.518, 7996 entries
>>>> 
>>>> Since I think we were convinced that this was intrinsic to the problem, and not a function of the Jacobian function, I am using my Jacobian.
>>>> 
>>>> Okay, I see no pattern in the fields. Lets plot these 2 vectors, -vec_view draw, and screenshot.
>>>> 
>>>>   Matt
>>>>  
>>>> -gideon
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150912/3e80b6e3/attachment.html>

From knepley at gmail.com  Sat Sep 12 13:57:40 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 12 Sep 2015 13:57:40 -0500
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <B81E8617-6EBB-45C2-8C3D-F7EC26E51215@gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
	<5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
	<CAMYG4G=uZxLZG8drNnfgKi7M98y+=FQXRAsdsZBbHwpcHFhRWw@mail.gmail.com>
	<A502B698-F0B6-4084-95A2-4D694DCF3CAB@gmail.com>
	<CAMYG4Gn6HdUim1Y1FY5=W4HCjO=LmVCYknm+xxnk_s=YXLbcMg@mail.gmail.com>
	<B81E8617-6EBB-45C2-8C3D-F7EC26E51215@gmail.com>
Message-ID: <CAMYG4Gkw-X6sDBHBSzhL9eS8Scaz+WSpzb26bRKTVj_Ois+c+A@mail.gmail.com>

On Sat, Sep 12, 2015 at 1:54 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> I?m not sure how good a problem this would be as a toy model, but, once
> this is all done, I?d be happy to share the code.
>
> I?ve got a few comments on this:
>
> 1.  The underlying problem is in the study of derivative nonlinear
> Schr?dinger equations - see http://arxiv.org/abs/1301.1048 for details of
> the equation.  The problem has features of the Airy equation, which as
> you?d guess, has exactly this problem of highly oscillatory structures
> which don?t get along with linear interpolation.
>
> 2.  The problem is complex valued with a real and imaginary components at
> each site on the mesh.  At the origin, the boundary conditions are
> Dirichlet on the imaginary component, and Neumann on the real component.
> This led me to make the value of the real component at the origin, u0, one
> of the three parameters in the first component of the DMComposite
> structure.  Consequently, (u0, 0), the values of the function at the
> origin, should be involved in the interpolation strategy.  Is there a way,
> through ghost points, for the current DM interpolation strategy to make use
> of these under refinement?  Does it use ghost point values?
>
> 3.  Is DMShellSetCreateInterpolation what I want to use here?  Where I
> would define my own interpolation strategy across dm?s?
>

Ah, crap. This breaks the current DMComposite interpolation (in packm.c),
which assumes that it is block diagonal. However, it produces
a MatNest object, so we could conceivably just stick in another block, but
that would not be general in any sense.

  Thanks,

     Matt


> -gideon
>
> On Sep 12, 2015, at 2:36 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Sat, Sep 12, 2015 at 1:26 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
>> Got it.  I wanted to follow up on something that I had been suspicious of
>> for some time.  I took my initial solution, the one which we want to apply
>> grid sequencing to, and did the following.  I loved into spicy, and
>> manually did grid refinement using SciPy, dumped to disk, and then ran that
>> through petsc, with the refined mesh.  When I say ?manual? here, I just
>> mean that I called the SciPy spline interplant commands on uniform meshes
>> that had twice the resolution, not that I manually picked out the
>> refinement points.  When I use a linear interpolant, and run my solver on
>> it, it fails miserably, just as the first grid sequence refinement does.
>> The numbers are a little different, but the residuals are still O(100).
>>
>> If, instead, I use a second order interpolant, it solves it 11
>> iterations, wiithout a problem, and the result is physically consistent.
>> While this may be two sides of the same coin, it makes me think that the
>> real challenge in my is the interpolation strategy used under grid
>> refinement.  Is there a way to, manually, I suppose, set the interpolation
>> that will be used under the grid sequence?
>>
>
> Excellent! And also somewhat bizarre. This would be great to analyze. Is
> there any bribe we could give you to
> convert this to a real PETSc example? It would be a nice way to show that
> dumping in high frequency energy
> when interpolating can be bad for coupled problems.
>
> Actual Solution: I think this will involve some coding. The quick and
> dirty way is to just stick the code you want
> into dainterp.c:DMCreateInterpolation_DA(). If that works, we add it to
> interptype and enable it from the command
> line. It would be nice to have a spline interpolant.
>
>   Thanks,
>
>      Matt
>
>
>> -gideon
>>
>> On Sep 12, 2015, at 7:14 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Fri, Sep 11, 2015 at 4:30 PM, Gideon Simpson <gideon.simpson at gmail.com
>> > wrote:
>>
>>> Are there any built in routines for freezing variables in SNES, or will
>>> that need to be handled by hand.
>>>
>>
>> We used to have this for KSP, but it looks like someone removed it. The
>> only thing left is KSPMonitorRange().
>> What we did is find the few largest residual elements, take a small halo
>> around them, project the problem to this
>> small space using MatGetSubMatrix() and VecScatter, solve, and VecScatter
>> back.
>>
>> We do not have this for nonlinear stuff (like many other things) because
>> there is no explicit matrix to manipulate,
>> and language support for computing only parts of the nonlinear function
>> is really weak. What we really want
>> is something like what Victor Eijkout was proposing a few years ago,
>> namely automatic discovery of index sets
>> for communication, in this case with main memory. We would need the
>> residual code, given an output set, to tell
>> us what input set is needed. Then we make a Scatter, select part of the
>> DM, and we could compute. Now it has
>> to be done by hand.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Also, I remain curious about the starting guess that the grid sequence
>>> uses during each refinement.  Is there a way to dump those to disk for
>>> inspection?
>>>
>>> -gideon
>>>
>>> On Sep 11, 2015, at 4:05 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>> On Fri, Sep 11, 2015 at 1:05 PM, Gideon Simpson <
>>> gideon.simpson at gmail.com> wrote:
>>>
>>>> Since the problem has not only the two components in the DM, but the
>>>> second component has 4 degrees of freedom per mesh point, I thought it best
>>>> to do the post processing separately.  See attached
>>>>
>>>
>>> So the whole thing is being controlled by 1 variable.
>>>
>>> We should try freezing everything else, and just solving that scalar
>>> equation I guess.
>>>
>>>    Matt
>>>
>>>
>>>> -gideon
>>>> <Screen Shot 2015-09-11 at 2.04.25 PM.png>
>>>>
>>>> On Sep 11, 2015, at 10:16 AM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>> On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <
>>>> gideon.simpson at gmail.com> wrote:
>>>>
>>>>> Following up on the previous thread, for my dm composite problem, I
>>>>> find that at the end of the first grid sequence,where it fails to converge,
>>>>> the distribution of the norms between the two pieces are:
>>>>>
>>>>>  39 SNES Function norm 2.253098577796e+02
>>>>>  40 SNES Function norm 2.253098577331e+02
>>>>>  41 SNES Function norm 2.253098577228e+02
>>>>>  42 SNES Function norm 2.253098577212e+02
>>>>>  43 SNES Function norm 2.253098577174e+02
>>>>>  44 SNES Function norm 2.253098577166e+02
>>>>>  45 SNES Function norm 2.253098577158e+02
>>>>>  46 SNES Function norm 2.253098577157e+02
>>>>>  47 SNES Function norm 2.253098577156e+02
>>>>>  48 SNES Function norm 2.253098577156e+02
>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH
>>>>> iterations 48
>>>>>  ||r|| = 225.31, 7999 entries
>>>>>  ||rp|| = 140.021, 3 entries
>>>>>  ||rQ|| = 176.518, 7996 entries
>>>>>
>>>>> Since I think we were convinced that this was intrinsic to the
>>>>> problem, and not a function of the Jacobian function, I am using my
>>>>> Jacobian.
>>>>>
>>>>
>>>> Okay, I see no pattern in the fields. Lets plot these 2 vectors,
>>>> -vec_view draw, and screenshot.
>>>>
>>>>   Matt
>>>>
>>>>
>>>>> -gideon
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150912/ed10f398/attachment-0001.html>

From bsmith at mcs.anl.gov  Sat Sep 12 15:43:55 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 12 Sep 2015 15:43:55 -0500
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <CAMYG4Gkw-X6sDBHBSzhL9eS8Scaz+WSpzb26bRKTVj_Ois+c+A@mail.gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
	<5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
	<CAMYG4G=uZxLZG8drNnfgKi7M98y+=FQXRAsdsZBbHwpcHFhRWw@mail.gmail.com>
	<A502B698-F0B6-4084-95A2-4D694DCF3CAB@gmail.com>
	<CAMYG4Gn6HdUim1Y1FY5=W4HCjO=LmVCYknm+xxnk_s=YXLbcMg@mail.gmail.com>
	<B81E8617-6EBB-45C2-8C3D-F7EC26E51215@gmail.com>
	<CAMYG4Gkw-X6sDBHBSzhL9eS8Scaz+WSpzb26bRKTVj_Ois+c+A@mail.gmail.com>
Message-ID: <A87185B3-A0EB-4000-A32E-5F1061335A17@mcs.anl.gov>


> On Sep 12, 2015, at 1:57 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Sat, Sep 12, 2015 at 1:54 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> I?m not sure how good a problem this would be as a toy model, but, once this is all done, I?d be happy to share the code.  
> 
> I?ve got a few comments on this:
> 
> 1.  The underlying problem is in the study of derivative nonlinear Schr?dinger equations - see http://arxiv.org/abs/1301.1048 for details of the equation.  The problem has features of the Airy equation, which as you?d guess, has exactly this problem of highly oscillatory structures which don?t get along with linear interpolation.
> 
> 2.  The problem is complex valued with a real and imaginary components at each site on the mesh.  At the origin, the boundary conditions are Dirichlet on the imaginary component, and Neumann on the real component.  This led me to make the value of the real component at the origin, u0, one of the three parameters in the first component of the DMComposite structure.  Consequently, (u0, 0), the values of the function at the origin, should be involved in the interpolation strategy.


    I do not understand. You have the real part of the solution at the origin of u0 stored in the "redundant" part of the DMComposite? What about the on the DMDA mesh? Wouldn't the real part also be there and with the same value? Thus when you interpolate with the solution on the DMDA you would still get the correct values "near" the origin? Hence there is no reason to "involve" the u0 stored in the redundant part in the interpolation?

  Barry

  
>  Is there a way, through ghost points, for the current DM interpolation strategy to make use of these under refinement?  Does it use ghost point values?
> 
> 3.  Is DMShellSetCreateInterpolation what I want to use here?  Where I would define my own interpolation strategy across dm?s?  
> 
> Ah, crap. This breaks the current DMComposite interpolation (in packm.c), which assumes that it is block diagonal. However, it produces
> a MatNest object, so we could conceivably just stick in another block, but that would not be general in any sense.
> 
>   Thanks,
> 
>      Matt
>  
> -gideon
> 
>> On Sep 12, 2015, at 2:36 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Sat, Sep 12, 2015 at 1:26 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> Got it.  I wanted to follow up on something that I had been suspicious of for some time.  I took my initial solution, the one which we want to apply grid sequencing to, and did the following.  I loved into spicy, and manually did grid refinement using SciPy, dumped to disk, and then ran that through petsc, with the refined mesh.  When I say ?manual? here, I just mean that I called the SciPy spline interplant commands on uniform meshes that had twice the resolution, not that I manually picked out the refinement points.  When I use a linear interpolant, and run my solver on it, it fails miserably, just as the first grid sequence refinement does.  The numbers are a little different, but the residuals are still O(100).
>> 
>> If, instead, I use a second order interpolant, it solves it 11 iterations, wiithout a problem, and the result is physically consistent.  While this may be two sides of the same coin, it makes me think that the real challenge in my is the interpolation strategy used under grid refinement.  Is there a way to, manually, I suppose, set the interpolation that will be used under the grid sequence?
>> 
>> Excellent! And also somewhat bizarre. This would be great to analyze. Is there any bribe we could give you to
>> convert this to a real PETSc example? It would be a nice way to show that dumping in high frequency energy
>> when interpolating can be bad for coupled problems.
>> 
>> Actual Solution: I think this will involve some coding. The quick and dirty way is to just stick the code you want
>> into dainterp.c:DMCreateInterpolation_DA(). If that works, we add it to interptype and enable it from the command
>> line. It would be nice to have a spline interpolant.
>> 
>>   Thanks,
>> 
>>      Matt
>>  
>> -gideon
>> 
>>> On Sep 12, 2015, at 7:14 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>> On Fri, Sep 11, 2015 at 4:30 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> Are there any built in routines for freezing variables in SNES, or will that need to be handled by hand.
>>> 
>>> We used to have this for KSP, but it looks like someone removed it. The only thing left is KSPMonitorRange().
>>> What we did is find the few largest residual elements, take a small halo around them, project the problem to this
>>> small space using MatGetSubMatrix() and VecScatter, solve, and VecScatter back.
>>> 
>>> We do not have this for nonlinear stuff (like many other things) because there is no explicit matrix to manipulate,
>>> and language support for computing only parts of the nonlinear function is really weak. What we really want
>>> is something like what Victor Eijkout was proposing a few years ago, namely automatic discovery of index sets
>>> for communication, in this case with main memory. We would need the residual code, given an output set, to tell
>>> us what input set is needed. Then we make a Scatter, select part of the DM, and we could compute. Now it has
>>> to be done by hand.
>>> 
>>>   Thanks,
>>> 
>>>      Matt
>>>  
>>> Also, I remain curious about the starting guess that the grid sequence uses during each refinement.  Is there a way to dump those to disk for inspection?
>>> 
>>> -gideon
>>> 
>>>> On Sep 11, 2015, at 4:05 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>> 
>>>> On Fri, Sep 11, 2015 at 1:05 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> Since the problem has not only the two components in the DM, but the second component has 4 degrees of freedom per mesh point, I thought it best to do the post processing separately.  See attached
>>>> 
>>>> So the whole thing is being controlled by 1 variable.
>>>> 
>>>> We should try freezing everything else, and just solving that scalar equation I guess.
>>>> 
>>>>    Matt
>>>>  
>>>> -gideon
>>>> <Screen Shot 2015-09-11 at 2.04.25 PM.png>
>>>> 
>>>>> On Sep 11, 2015, at 10:16 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>> 
>>>>> On Fri, Sep 11, 2015 at 9:08 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> Following up on the previous thread, for my dm composite problem, I find that at the end of the first grid sequence,where it fails to converge, the distribution of the norms between the two pieces are:
>>>>> 
>>>>>  39 SNES Function norm 2.253098577796e+02 
>>>>>  40 SNES Function norm 2.253098577331e+02 
>>>>>  41 SNES Function norm 2.253098577228e+02 
>>>>>  42 SNES Function norm 2.253098577212e+02 
>>>>>  43 SNES Function norm 2.253098577174e+02 
>>>>>  44 SNES Function norm 2.253098577166e+02 
>>>>>  45 SNES Function norm 2.253098577158e+02 
>>>>>  46 SNES Function norm 2.253098577157e+02 
>>>>>  47 SNES Function norm 2.253098577156e+02 
>>>>>  48 SNES Function norm 2.253098577156e+02 
>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 48
>>>>>  ||r|| = 225.31, 7999 entries
>>>>>  ||rp|| = 140.021, 3 entries
>>>>>  ||rQ|| = 176.518, 7996 entries
>>>>> 
>>>>> Since I think we were convinced that this was intrinsic to the problem, and not a function of the Jacobian function, I am using my Jacobian.
>>>>> 
>>>>> Okay, I see no pattern in the fields. Lets plot these 2 vectors, -vec_view draw, and screenshot.
>>>>> 
>>>>>   Matt
>>>>>  
>>>>> -gideon
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From gideon.simpson at gmail.com  Sat Sep 12 16:38:35 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Sat, 12 Sep 2015 17:38:35 -0400
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <A87185B3-A0EB-4000-A32E-5F1061335A17@mcs.anl.gov>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
	<5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
	<CAMYG4G=uZxLZG8drNnfgKi7M98y+=FQXRAsdsZBbHwpcHFhRWw@mail.gmail.com>
	<A502B698-F0B6-4084-95A2-4D694DCF3CAB@gmail.com>
	<CAMYG4Gn6HdUim1Y1FY5=W4HCjO=LmVCYknm+xxnk_s=YXLbcMg@mail.gmail.com>
	<B81E8617-6EBB-45C2-8C3D-F7EC26E51215@gmail.com>
	<CAMYG4Gkw-X6sDBHBSzhL9eS8Scaz+WSpzb26bRKTVj_Ois+c+A@mail.gmail.com>
	<A87185B3-A0EB-4000-A32E-5F1061335A17@mcs.anl.gov>
Message-ID: <055A5A63-38B4-4838-BE1D-55BF1BA69433@gmail.com>


> On Sep 12, 2015, at 4:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>    I do not understand. You have the real part of the solution at the origin of u0 stored in the "redundant" part of the DMComposite? What about the on the DMDA mesh? Wouldn't the real part also be there and with the same value? Thus when you interpolate with the solution on the DMDA you would still get the correct values "near" the origin? Hence there is no reason to "involve" the u0 stored in the redundant part in the interpolation?
> 
>  Barry

Suppose my coarse mesh is x1, x2, x3,?, and x0 = 0, corresponds to the origin.  At x0=0, my constraints are that u?(0) = 0, and v(0) = 0.  So u0, the value at the origin, is an unknown that needs to be solved for.  Now, when I do grid sequencing, I ought to be adding a point between x0 and x1, but I don?t, because x0 isn?t part of the DMDA.  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150912/7c70b7e9/attachment.html>

From bsmith at mcs.anl.gov  Sat Sep 12 16:50:43 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 12 Sep 2015 16:50:43 -0500
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <055A5A63-38B4-4838-BE1D-55BF1BA69433@gmail.com>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
	<5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
	<CAMYG4G=uZxLZG8drNnfgKi7M98y+=FQXRAsdsZBbHwpcHFhRWw@mail.gmail.com>
	<A502B698-F0B6-4084-95A2-4D694DCF3CAB@gmail.com>
	<CAMYG4Gn6HdUim1Y1FY5=W4HCjO=LmVCYknm+xxnk_s=YXLbcMg@mail.gmail.com>
	<B81E8617-6EBB-45C2-8C3D-F7EC26E51215@gmail.com>
	<CAMYG4Gkw-X6sDBHBSzhL9eS8Scaz+WSpzb26bRKTVj_Ois+c+A@mail.gmail.com>
	<A87185B3-A0EB-4000-A32E-5F1061335A17@mcs.anl.gov>
	<055A5A63-38B4-4838-BE1D-55BF1BA69433@gmail.com>
Message-ID: <B75C7744-852B-4F5F-BEDE-62FF60B1E154@mcs.anl.gov>


> On Sep 12, 2015, at 4:38 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> 
>> On Sep 12, 2015, at 4:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>>    I do not understand. You have the real part of the solution at the origin of u0 stored in the "redundant" part of the DMComposite? What about the on the DMDA mesh? Wouldn't the real part also be there and with the same value? Thus when you interpolate with the solution on the DMDA you would still get the correct values "near" the origin? Hence there is no reason to "involve" the u0 stored in the redundant part in the interpolation?
>> 
>>  Barry
> 

   Oh, you are doing a 1d problem! 

   Why put that x0 in a different place then the "interior points"? I would never do that because you are right the whole interpolation business is messed up.  Can you reorganize things so that x0 is just included in the DMDA? If not, why not and we'll fix them. It would be insanely cumbersome and totally unnecessary IMHO to build a complicated construct to handle the interpolation with the x0 in a different place. Jeepers, of course the interpolation won't work.

  Barry



> Suppose my coarse mesh is x1, x2, x3,?, and x0 = 0, corresponds to the origin.  At x0=0, my constraints are that u?(0) = 0, and v(0) = 0.  So u0, the value at the origin, is an unknown that needs to be solved for.  Now, when I do grid sequencing, I ought to be adding a point between x0 and x1, but I don?t, because x0 isn?t part of the DMDA.  


From rlmackie862 at gmail.com  Mon Sep 14 13:15:34 2015
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Mon, 14 Sep 2015 11:15:34 -0700
Subject: [petsc-users] strange FPE divide by zero
Message-ID: <06AF5549-8900-4696-A2D8-A62731EB7574@gmail.com>

I?ve run into a strange error, which is that when I compile my Fortran code with -ffpe-trap=invalid it bombs out and gives the backtrace below. If I don?t include the ffpe-trap switch, the code runs fine and gives the expected results.

I?ve even run the code through Valgrind, and no issues were found.

I don?t know if this is just something strange with my matrix, or this is a bug somewhere in the PETSc code.

I do have a small test program and a large binary matrix and vector file if you want to test.


Randy


[9]PETSC ERROR: ------------------------------------------------------------------------
[9]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
[9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[9]PETSC ERROR: [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] LAPACKgesvd line 44 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/impls/gmres/gmreig.c
[0]PETSC ERROR: [0] KSPComputeExtremeSingularValues_GMRES line 24 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/impls/gmres/gmreig.c
[0]PETSC ERROR: [0] KSPComputeExtremeSingularValues line 51 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: likely location of problem given in stack below
[2]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[2]PETSC ERROR:       INSTEAD the line number of the start of the function
[2]PETSC ERROR:       is given.
[2]PETSC ERROR: [2] LAPACKgesvd line 44 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/impls/gmres/gmreig.c
[2]PETSC ERROR: [2] KSPComputeExtremeSingularValues_GMRES line 24 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/impls/gmres/gmreig.c
[2]PETSC ERROR: [2] KSPComputeExtremeSingularValues line 51 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: [2] PCGAMGOptProlongator_AGG line 1187 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/pc/impls/gamg/agg.c
[2]PETSC ERROR: [2] PCSetUp_GAMG line 468 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/pc/impls/gamg/gamg.c
[2]PETSC ERROR: [2] PCSetUp line 944 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/pc/interface/precon.c
[2]PETSC ERROR: [2] KSPSetUp line 247 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: [2] KSPSolve line 510 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Signal received
[5]PETSC ERROR: [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.


This happens using either mpich or mvapich. The mvapich backtrace shows this:

[compute-1-11.local:mpi_rank_6][print_backtrace]   5: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(+0x268d24) [0x2ba2fac58d24]
[compute-1-11.local:mpi_rank_6][print_backtrace]   6: /lib64/libc.so.6() [0x38af6329a0]
[compute-1-11.local:mpi_rank_6][print_backtrace]   7: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(ieeeck_+0x1e9) [0x2ba2fc0e7b55]
[compute-1-11.local:mpi_rank_6][print_backtrace]   8: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(ilaenv_+0x1b17) [0x2ba2fc0baa77]
[compute-1-11.local:mpi_rank_6][print_backtrace]   9: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(dlasq2_+0x9a2) [0x2ba2fc0dd53e]
[compute-1-11.local:mpi_rank_6][print_backtrace]  10: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(dlasq1_+0x45d) [0x2ba2fc0dc941]
[compute-1-11.local:mpi_rank_6][print_backtrace]  11: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(zbdsqr_+0x33e) [0x2ba2fc0bb42a]
[compute-1-11.local:mpi_rank_6][print_backtrace]  12: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(zgesvd_+0x9091) [0x2ba2fc089445]
[compute-1-11.local:mpi_rank_6][print_backtrace]  13: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(KSPComputeExtremeSingularValues_GMRES+0x7e0) [0x2ba2fbcf77fd]
[compute-1-11.local:mpi_rank_6][print_backtrace]  14: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(KSPComputeExtremeSingularValues+0x4ec) [0x2ba2fbd78ea6]
[compute-1-11.local:mpi_rank_6][print_backtrace]  15: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(PCGAMGOptProlongator_AGG+0x111a) [0x2ba2fbbb28f9]
[compute-1-11.local:mpi_rank_6][print_backtrace]  16: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(PCSetUp_GAMG+0x1565) [0x2ba2fbb9995b]
[compute-1-11.local:mpi_rank_6][print_backtrace]  17: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(PCSetUp+0xc6b) [0x2ba2fbcb06b3]
[compute-1-11.local:mpi_rank_6][print_backtrace]  18: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(KSPSetUp+0x14e5) [0x2ba2fbd7bac3]
[compute-1-11.local:mpi_rank_6][print_backtrace]  19: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(KSPSolve+0xc31) [0x2ba2fbd7d70a]
[compute-1-11.local:mpi_rank_6][print_backtrace]  20: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(kspsolve_+0x3e) [0x2ba2fbdbba40]
[compute-1-11.local:mpi_rank_6][print_backtrace]  21: ./test() [0x401425]
[compute-1-11.local:mpi_rank_6][print_backtrace]  22: ./test() [0x4014ca]




From bsmith at mcs.anl.gov  Mon Sep 14 13:53:18 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 14 Sep 2015 13:53:18 -0500
Subject: [petsc-users] strange FPE divide by zero
In-Reply-To: <06AF5549-8900-4696-A2D8-A62731EB7574@gmail.com>
References: <06AF5549-8900-4696-A2D8-A62731EB7574@gmail.com>
Message-ID: <CF088E97-47B2-4E20-BADF-290C93B3F010@mcs.anl.gov>


  Randy,

     This is actually expected behavior. LAPACK was designed and implemented treating Inf is a valid numerical result, thus LAPACK will generate Inf internally as it computes and resulting in correct answers. 

     Of course this can be annoying because there is no easy way to distinguish between "reasonable" floating point exceptions (like Inf in LAPACK) and bad floating point exceptions caused by bugs in code. In PETSc when you run with -fp_trap on we "turn off" the trapping during LAPACK calls those "ok" floating point exceptions are ignored but it will still trap other floating point exceptions that are likely bugs.

   Barry


> On Sep 14, 2015, at 1:15 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> 
> I?ve run into a strange error, which is that when I compile my Fortran code with -ffpe-trap=invalid it bombs out and gives the backtrace below. If I don?t include the ffpe-trap switch, the code runs fine and gives the expected results.
> 
> I?ve even run the code through Valgrind, and no issues were found.
> 
> I don?t know if this is just something strange with my matrix, or this is a bug somewhere in the PETSc code.
> 
> I do have a small test program and a large binary matrix and vector file if you want to test.
> 
> 
> Randy
> 
> 
> [9]PETSC ERROR: ------------------------------------------------------------------------
> [9]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
> [9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [9]PETSC ERROR: [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] LAPACKgesvd line 44 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/impls/gmres/gmreig.c
> [0]PETSC ERROR: [0] KSPComputeExtremeSingularValues_GMRES line 24 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/impls/gmres/gmreig.c
> [0]PETSC ERROR: [0] KSPComputeExtremeSingularValues line 51 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: likely location of problem given in stack below
> [2]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [2]PETSC ERROR:       INSTEAD the line number of the start of the function
> [2]PETSC ERROR:       is given.
> [2]PETSC ERROR: [2] LAPACKgesvd line 44 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/impls/gmres/gmreig.c
> [2]PETSC ERROR: [2] KSPComputeExtremeSingularValues_GMRES line 24 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/impls/gmres/gmreig.c
> [2]PETSC ERROR: [2] KSPComputeExtremeSingularValues line 51 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: [2] PCGAMGOptProlongator_AGG line 1187 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/pc/impls/gamg/agg.c
> [2]PETSC ERROR: [2] PCSetUp_GAMG line 468 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/pc/impls/gamg/gamg.c
> [2]PETSC ERROR: [2] PCSetUp line 944 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/pc/interface/precon.c
> [2]PETSC ERROR: [2] KSPSetUp line 247 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: [2] KSPSolve line 510 /state/std2/FEMI/PETSc/petsc-3.6.1/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [2]PETSC ERROR: Signal received
> [5]PETSC ERROR: [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> 
> 
> This happens using either mpich or mvapich. The mvapich backtrace shows this:
> 
> [compute-1-11.local:mpi_rank_6][print_backtrace]   5: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(+0x268d24) [0x2ba2fac58d24]
> [compute-1-11.local:mpi_rank_6][print_backtrace]   6: /lib64/libc.so.6() [0x38af6329a0]
> [compute-1-11.local:mpi_rank_6][print_backtrace]   7: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(ieeeck_+0x1e9) [0x2ba2fc0e7b55]
> [compute-1-11.local:mpi_rank_6][print_backtrace]   8: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(ilaenv_+0x1b17) [0x2ba2fc0baa77]
> [compute-1-11.local:mpi_rank_6][print_backtrace]   9: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(dlasq2_+0x9a2) [0x2ba2fc0dd53e]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  10: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(dlasq1_+0x45d) [0x2ba2fc0dc941]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  11: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(zbdsqr_+0x33e) [0x2ba2fc0bb42a]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  12: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(zgesvd_+0x9091) [0x2ba2fc089445]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  13: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(KSPComputeExtremeSingularValues_GMRES+0x7e0) [0x2ba2fbcf77fd]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  14: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(KSPComputeExtremeSingularValues+0x4ec) [0x2ba2fbd78ea6]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  15: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(PCGAMGOptProlongator_AGG+0x111a) [0x2ba2fbbb28f9]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  16: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(PCSetUp_GAMG+0x1565) [0x2ba2fbb9995b]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  17: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(PCSetUp+0xc6b) [0x2ba2fbcb06b3]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  18: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(KSPSetUp+0x14e5) [0x2ba2fbd7bac3]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  19: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(KSPSolve+0xc31) [0x2ba2fbd7d70a]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  20: /state/std2/FEMI/PETSc/petsc-3.6.1/linux-gfortran-debug/lib/libpetsc.so.3.6(kspsolve_+0x3e) [0x2ba2fbdbba40]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  21: ./test() [0x401425]
> [compute-1-11.local:mpi_rank_6][print_backtrace]  22: ./test() [0x4014ca]
> 
> 
> 


From gideon.simpson at gmail.com  Mon Sep 14 19:41:22 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 14 Sep 2015 20:41:22 -0400
Subject: [petsc-users] conditioning of snes with dmcomposite & grid
	sequencing
In-Reply-To: <B75C7744-852B-4F5F-BEDE-62FF60B1E154@mcs.anl.gov>
References: <3C9337A8-7F09-4B52-ADA5-CAEF50FCCD99@gmail.com>
	<CAMYG4G=njTF8Vb8PvGk-6SK_BprLcOAxssy7-jRR7gNOiT=s=w@mail.gmail.com>
	<D2397A19-7DF2-4A16-B6A0-42B13AA68007@gmail.com>
	<CAMYG4G=Cfq-6a_xL5Eqn6=H+hdywd3qXFfd6ZSk1qmsaPB5HxA@mail.gmail.com>
	<5A4A3BD7-1F08-487E-9023-388E9148A740@gmail.com>
	<CAMYG4G=uZxLZG8drNnfgKi7M98y+=FQXRAsdsZBbHwpcHFhRWw@mail.gmail.com>
	<A502B698-F0B6-4084-95A2-4D694DCF3CAB@gmail.com>
	<CAMYG4Gn6HdUim1Y1FY5=W4HCjO=LmVCYknm+xxnk_s=YXLbcMg@mail.gmail.com>
	<B81E8617-6EBB-45C2-8C3D-F7EC26E51215@gmail.com>
	<CAMYG4Gkw-X6sDBHBSzhL9eS8Scaz+WSpzb26bRKTVj_Ois+c+A@mail.gmail.com>
	<A87185B3-A0EB-4000-A32E-5F1061335A17@mcs.anl.gov>
	<055A5A63-38B4-4838-BE1D-55BF1BA69433@gmail.com>
	<B75C7744-852B-4F5F-BEDE-62FF60B1E154@mcs.anl.gov>
Message-ID: <2B1F8C2B-D6A3-4052-B69B-9A39DFFF58BC@gmail.com>

Ok, I have been able to recast so that the components are properly separated and grid sequencing appears to work.  
-gideon

> On Sep 12, 2015, at 5:50 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>> 
>> On Sep 12, 2015, at 4:38 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> 
>>> On Sep 12, 2015, at 4:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>>   I do not understand. You have the real part of the solution at the origin of u0 stored in the "redundant" part of the DMComposite? What about the on the DMDA mesh? Wouldn't the real part also be there and with the same value? Thus when you interpolate with the solution on the DMDA you would still get the correct values "near" the origin? Hence there is no reason to "involve" the u0 stored in the redundant part in the interpolation?
>>> 
>>> Barry
>> 
> 
>   Oh, you are doing a 1d problem! 
> 
>   Why put that x0 in a different place then the "interior points"? I would never do that because you are right the whole interpolation business is messed up.  Can you reorganize things so that x0 is just included in the DMDA? If not, why not and we'll fix them. It would be insanely cumbersome and totally unnecessary IMHO to build a complicated construct to handle the interpolation with the x0 in a different place. Jeepers, of course the interpolation won't work.
> 
>  Barry
> 
> 
> 
>> Suppose my coarse mesh is x1, x2, x3,?, and x0 = 0, corresponds to the origin.  At x0=0, my constraints are that u?(0) = 0, and v(0) = 0.  So u0, the value at the origin, is an unknown that needs to be solved for.  Now, when I do grid sequencing, I ought to be adding a point between x0 and x1, but I don?t, because x0 isn?t part of the DMDA.  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150914/4a101c5d/attachment.html>

From david.knezevic at akselos.com  Tue Sep 15 04:30:30 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Tue, 15 Sep 2015 17:30:30 +0800
Subject: [petsc-users] Automatically re-solving after MUMPS error
Message-ID: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>

In some cases, I get MUMPS error -9, i.e.:
[2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFO(1)=-9, INFO(2)=98927

This is easily fixed by re-running the executable with -mat_mumps_icntl_14
on the commandline.

However, I would like to update my code in order to do this automatically,
i.e. detect the -9 error and re-run with the appropriate option. Is there a
recommended way to do this? It seems to me that I could do this with a
PETSc error handler (e.g. PetscPushErrorHandler) in order to call a
function that sets the appropriate option and solves again, is that right?
Are there any examples that illustrate this type of thing?

Thanks,
David

 <http://www.akselos.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150915/b2415595/attachment-0001.html>

From knepley at gmail.com  Tue Sep 15 06:29:09 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 15 Sep 2015 06:29:09 -0500
Subject: [petsc-users] Automatically re-solving after MUMPS error
In-Reply-To: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
References: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
Message-ID: <CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>

On Tue, Sep 15, 2015 at 4:30 AM, David Knezevic <david.knezevic at akselos.com>
wrote:

> In some cases, I get MUMPS error -9, i.e.:
> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFO(1)=-9, INFO(2)=98927
>
> This is easily fixed by re-running the executable with -mat_mumps_icntl_14
> on the commandline.
>
> However, I would like to update my code in order to do this automatically,
> i.e. detect the -9 error and re-run with the appropriate option. Is there a
> recommended way to do this? It seems to me that I could do this with a
> PETSc error handler (e.g. PetscPushErrorHandler) in order to call a
> function that sets the appropriate option and solves again, is that right?
> Are there any examples that illustrate this type of thing?
>

I would not use the error handler. I would just check the ierr return code
from the solver. I think you need the
INFO output, for which you can use MatMumpsGetInfo().

  Thanks,

    Matt


> Thanks,
> David
>
>  <http://www.akselos.com>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150915/f7e7d0c9/attachment.html>

From david.knezevic at akselos.com  Tue Sep 15 06:47:30 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Tue, 15 Sep 2015 19:47:30 +0800
Subject: [petsc-users] Automatically re-solving after MUMPS error
In-Reply-To: <CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>
References: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
	<CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>
Message-ID: <CAJCWK9BihEkqFefFMPjQNBj5gzp6Zfh-r_gdh=Mh+SL5GJcD5A@mail.gmail.com>

On Tue, Sep 15, 2015 at 7:29 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Sep 15, 2015 at 4:30 AM, David Knezevic <
> david.knezevic at akselos.com> wrote:
>
>> In some cases, I get MUMPS error -9, i.e.:
>> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
>> INFO(1)=-9, INFO(2)=98927
>>
>> This is easily fixed by re-running the executable with
>> -mat_mumps_icntl_14 on the commandline.
>>
>> However, I would like to update my code in order to do this
>> automatically, i.e. detect the -9 error and re-run with the appropriate
>> option. Is there a recommended way to do this? It seems to me that I could
>> do this with a PETSc error handler (e.g. PetscPushErrorHandler) in order to
>> call a function that sets the appropriate option and solves again, is that
>> right? Are there any examples that illustrate this type of thing?
>>
>
> I would not use the error handler. I would just check the ierr return code
> from the solver. I think you need the
> INFO output, for which you can use MatMumpsGetInfo().
>


OK, that sounds good (and much simpler than what I had in mind), thanks for
the help!

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150915/bb7cf24c/attachment.html>

From erand at mek.dtu.dk  Tue Sep 15 14:35:30 2015
From: erand at mek.dtu.dk (Erik Andreassen)
Date: Tue, 15 Sep 2015 19:35:30 +0000
Subject: [petsc-users] Number of levels in PCGAMG
Message-ID: <300719361A150149BDBDF898D5BF89CE063CB350@ait-pex01mbx01.win.dtu.dk>

Hi,

I would like to know how to change the number of multigrid levels in AMG. I have tried PCMGSetLevels, which does not work, and I suspect that the PC is not recognized as a multigrid preconditioner, because I'm PCGAMG as the preconditioner to the coarse level solver of a PCMG. However, I do not understand why it is not recognized as a multigrid preconditioner.

Please let me know if the above is unclear,
Erik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150915/1f4c46db/attachment.html>

From jed at jedbrown.org  Tue Sep 15 15:01:52 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 15 Sep 2015 14:01:52 -0600
Subject: [petsc-users] Number of levels in PCGAMG
In-Reply-To: <300719361A150149BDBDF898D5BF89CE063CB350@ait-pex01mbx01.win.dtu.dk>
References: <300719361A150149BDBDF898D5BF89CE063CB350@ait-pex01mbx01.win.dtu.dk>
Message-ID: <87io7bwixb.fsf@jedbrown.org>

Erik Andreassen <erand at mek.dtu.dk> writes:

> Hi,
>
> I would like to know how to change the number of multigrid levels in AMG. I have tried PCMGSetLevels, which does not work, and I suspect that the PC is not recognized as a multigrid preconditioner, because I'm PCGAMG as the preconditioner to the coarse level solver of a PCMG. However, I do not understand why it is not recognized as a multigrid preconditioner.

PCGAMGSetNlevels
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150915/4f0f06b3/attachment.pgp>

From erand at mek.dtu.dk  Tue Sep 15 15:07:53 2015
From: erand at mek.dtu.dk (Erik Andreassen)
Date: Tue, 15 Sep 2015 20:07:53 +0000
Subject: [petsc-users] Number of levels in PCGAMG
In-Reply-To: <87io7bwixb.fsf@jedbrown.org>
References: <300719361A150149BDBDF898D5BF89CE063CB350@ait-pex01mbx01.win.dtu.dk>,
	<87io7bwixb.fsf@jedbrown.org>
Message-ID: <300719361A150149BDBDF898D5BF89CE063CB367@ait-pex01mbx01.win.dtu.dk>

Thanks!
________________________________________
Fra: Jed Brown [jed at jedbrown.org]
Sendt: 15. september 2015 22:01
Til: Erik Andreassen; petsc-users at mcs.anl.gov
Emne: Re: [petsc-users] Number of levels in PCGAMG

Erik Andreassen <erand at mek.dtu.dk> writes:

> Hi,
>
> I would like to know how to change the number of multigrid levels in AMG. I have tried PCMGSetLevels, which does not work, and I suspect that the PC is not recognized as a multigrid preconditioner, because I'm PCGAMG as the preconditioner to the coarse level solver of a PCMG. However, I do not understand why it is not recognized as a multigrid preconditioner.

PCGAMGSetNlevels

From mfadams at lbl.gov  Tue Sep 15 15:33:33 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 15 Sep 2015 16:33:33 -0400
Subject: [petsc-users] Number of levels in PCGAMG
In-Reply-To: <300719361A150149BDBDF898D5BF89CE063CB367@ait-pex01mbx01.win.dtu.dk>
References: <300719361A150149BDBDF898D5BF89CE063CB350@ait-pex01mbx01.win.dtu.dk>
	<87io7bwixb.fsf@jedbrown.org>
	<300719361A150149BDBDF898D5BF89CE063CB367@ait-pex01mbx01.win.dtu.dk>
Message-ID: <CADOhEh7qWop1vinm17q_1Nzdnfm+0sO5hCeDrz1SrZC3+vrLAw@mail.gmail.com>

BTW, it is recommended that you let GAMG figure out how many levels.  THere
are parameters to guild this and they are not as problem dependant.  But if
you really know how many levels you want, know what you are doing, then go
for it.
Mark

On Tue, Sep 15, 2015 at 4:07 PM, Erik Andreassen <erand at mek.dtu.dk> wrote:

> Thanks!
> ________________________________________
> Fra: Jed Brown [jed at jedbrown.org]
> Sendt: 15. september 2015 22:01
> Til: Erik Andreassen; petsc-users at mcs.anl.gov
> Emne: Re: [petsc-users] Number of levels in PCGAMG
>
> Erik Andreassen <erand at mek.dtu.dk> writes:
>
> > Hi,
> >
> > I would like to know how to change the number of multigrid levels in
> AMG. I have tried PCMGSetLevels, which does not work, and I suspect that
> the PC is not recognized as a multigrid preconditioner, because I'm PCGAMG
> as the preconditioner to the coarse level solver of a PCMG. However, I do
> not understand why it is not recognized as a multigrid preconditioner.
>
> PCGAMGSetNlevels
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150915/08bf5764/attachment.html>

From a.croucher at auckland.ac.nz  Tue Sep 15 21:05:08 2015
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Wed, 16 Sep 2015 14:05:08 +1200
Subject: [petsc-users] block matrix in serial
Message-ID: <55F8CE54.5010707@auckland.ac.nz>

hi

I have a test code (attached) that sets up a finite volume mesh using 
DMPlex, with 2 degrees of freedom per cell.

I then create a matrix using DMCreateMatrix(), having used 
DMSetMatType() to set the matrix type to MATBAIJ or MATMPIBAIJ, to take 
advantage of the block structure.

This works OK and gives me the expected matrix structure when I run on > 
1 processor, but neither MATBAIJ or MATMPIBAIJ works if I run it in serial:

[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Blocksize of layout 2 must match that of mapping 1
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.6.1-811-gaf4f2ea GIT 
Date: 2015-08-18 17:44:25 -0500
[0]PETSC ERROR: ./testmat on a linux-gnu-c-opt named des108 by acro018 
Wed Sep 16 13:51:27 2015
[0]PETSC ERROR: Configure options --download-netcdf --download-exodusii 
--with-hdf5-dir=/usr --download-triangle --download-ptscotch -
-download-chaco
[0]PETSC ERROR: #1 PetscLayoutSetISLocalToGlobalMapping() line 248 in 
/home/acro018/software/PETSc/code/src/vec/is/utils/pmap.c
[0]PETSC ERROR: #2 MatSetLocalToGlobalMapping() line 1876 in 
/home/acro018/software/PETSc/code/src/mat/interface/matrix.c
[0]PETSC ERROR: #3 DMCreateMatrix_Plex() line 853 in 
/home/acro018/software/PETSc/code/src/dm/impls/plex/plex.c
[0]PETSC ERROR: #4 DMCreateMatrix() line 961 in 
/home/acro018/software/PETSc/code/src/dm/interface/dm.c
--------------------------------------------------------------------------

It only works in serial if I use MATAIJ.

Is there a reason why the block matrix types don't appear to work in 
serial, at least in this case? (I'm running the PETSc 'next' branch.)

- Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 84611

-------------- next part --------------
A non-text attachment was scrubbed...
Name: testmat.F90
Type: text/x-fortran
Size: 1352 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150916/4230723d/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: col10.exo
Type: application/octet-stream
Size: 3516 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150916/4230723d/attachment.obj>

From knepley at gmail.com  Wed Sep 16 06:40:37 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Sep 2015 06:40:37 -0500
Subject: [petsc-users] block matrix in serial
In-Reply-To: <55F8CE54.5010707@auckland.ac.nz>
References: <55F8CE54.5010707@auckland.ac.nz>
Message-ID: <CAMYG4G=i2O7nFpaVdrqqyYoEVtzHAyRt-MQmRKcfE2c-MWoezQ@mail.gmail.com>

On Tue, Sep 15, 2015 at 9:05 PM, Adrian Croucher <a.croucher at auckland.ac.nz>
wrote:

> hi
>
> I have a test code (attached) that sets up a finite volume mesh using
> DMPlex, with 2 degrees of freedom per cell.
>
> I then create a matrix using DMCreateMatrix(), having used DMSetMatType()
> to set the matrix type to MATBAIJ or MATMPIBAIJ, to take advantage of the
> block structure.
>
> This works OK and gives me the expected matrix structure when I run on > 1
> processor, but neither MATBAIJ or MATMPIBAIJ works if I run it in serial:
>

Plex should automatically discover the block size from the Section. If not,
it uses block size 1. I have to look
at the example to see why the discovery is not working. Do you have any
constraints?

  Thanks,

    Matt


> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: Blocksize of layout 2 must match that of mapping 1
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.6.1-811-gaf4f2ea GIT
> Date: 2015-08-18 17:44:25 -0500
> [0]PETSC ERROR: ./testmat on a linux-gnu-c-opt named des108 by acro018 Wed
> Sep 16 13:51:27 2015
> [0]PETSC ERROR: Configure options --download-netcdf --download-exodusii
> --with-hdf5-dir=/usr --download-triangle --download-ptscotch -
> -download-chaco
> [0]PETSC ERROR: #1 PetscLayoutSetISLocalToGlobalMapping() line 248 in
> /home/acro018/software/PETSc/code/src/vec/is/utils/pmap.c
> [0]PETSC ERROR: #2 MatSetLocalToGlobalMapping() line 1876 in
> /home/acro018/software/PETSc/code/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 DMCreateMatrix_Plex() line 853 in
> /home/acro018/software/PETSc/code/src/dm/impls/plex/plex.c
> [0]PETSC ERROR: #4 DMCreateMatrix() line 961 in
> /home/acro018/software/PETSc/code/src/dm/interface/dm.c
> --------------------------------------------------------------------------
>
> It only works in serial if I use MATAIJ.
>
> Is there a reason why the block matrix types don't appear to work in
> serial, at least in this case? (I'm running the PETSc 'next' branch.)
>
> - Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> University of Auckland, New Zealand
> email: a.croucher at auckland.ac.nz
> tel: +64 (0)9 923 84611
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150916/cca49086/attachment-0001.html>

From lawrence.mitchell at imperial.ac.uk  Wed Sep 16 07:18:57 2015
From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell)
Date: Wed, 16 Sep 2015 13:18:57 +0100
Subject: [petsc-users] block matrix in serial
In-Reply-To: <CAMYG4G=i2O7nFpaVdrqqyYoEVtzHAyRt-MQmRKcfE2c-MWoezQ@mail.gmail.com>
References: <55F8CE54.5010707@auckland.ac.nz>
	<CAMYG4G=i2O7nFpaVdrqqyYoEVtzHAyRt-MQmRKcfE2c-MWoezQ@mail.gmail.com>
Message-ID: <55F95E31.6010404@imperial.ac.uk>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 16/09/15 12:40, Matthew Knepley wrote:
> On Tue, Sep 15, 2015 at 9:05 PM, Adrian Croucher 
> <a.croucher at auckland.ac.nz <mailto:a.croucher at auckland.ac.nz>>
> wrote:
> 
> hi
> 
> I have a test code (attached) that sets up a finite volume mesh 
> using DMPlex, with 2 degrees of freedom per cell.
> 
> I then create a matrix using DMCreateMatrix(), having used 
> DMSetMatType() to set the matrix type to MATBAIJ or MATMPIBAIJ, to 
> take advantage of the block structure.
> 
> This works OK and gives me the expected matrix structure when I
> run on > 1 processor, but neither MATBAIJ or MATMPIBAIJ works if I
> run it in serial:
> 
> 
> Plex should automatically discover the block size from the Section.
> If not, it uses block size 1. I have to look at the example to see
> why the discovery is not working. Do you have any constraints?

It looks like block discovery in parallel effectively always
determines a block size of 1.  Running with -mat_view ::ascii_info gives:

Mat Object: 2 MPI processes
  type: mpibaij
  rows=20, cols=20
  total: nonzeros=112, allocated nonzeros=112
  total number of mallocs used during MatSetValues calls =0
      block size is 1
                    ^^^

The block size discovery does this:

        for (p = pStart; p < pEnd; ++p) {
          ierr = PetscSectionGetDof(sectionGlobal, p, &dof);CHKERRQ(ierr);
          ierr = PetscSectionGetConstraintDof(sectionGlobal, p,
&cdof);CHKERRQ(ierr);
          if (dof-cdof) {
            if (bs < 0) {
              bs = dof-cdof;
            } else if (bs != dof-cdof) {
              /* Layout does not admit a pointwise block size */
              bs = 1;
              break;
            }
          }
        }

In parallel, some of the dofs are remote (encoded as negative).  So we
run through seeing (dof - cdof) == 2, until we hit a "remote" point at
when we see (dof - cdof) != 2 and then we break out and set bs = 1.

In serial, we correctly determine bs == 2.  But then
DMGetLocalToGlobalMapping always does

      ierr = ISLocalToGlobalMappingCreate(PETSC_COMM_SELF, 1,size,
ltog, PETSC_OWN_POINTER, &dm->ltogmap);CHKERRQ(ierr);


i.e. is set with block size == 1.

So there are two bugs here.

1. In parallel, block size discovery in Mat creation has gone wrong
2. (Always), the lgmap has block size of 1, irrespective of the
discovered block size from the section.

Lawrence


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJV+V4tAAoJECOc1kQ8PEYvqO4IAN4+oIgtBevvDAughPgUVOzq
kESJkb0Bx4a7+y47IPsY/SOuOMjVgErz2SO3tGd7+K/U5fstojJbIC7zZqPIERn0
S68lH3s+y1pVqmcIMFIorz6is+u46M2xIGS6MLe6d0DluslyThaaA7lMhuIvJkIX
FC8QmtqCsIvHv10VuNll/81UJ3pXSZ+E81+Rs6pBhoGMCnRSXdMgFEtfWBBGL2JR
byEOAueTmY+YZXJ5JINxsHG1C1lep5wyYfAERDiRaD9bhCsE4mf9z/yFdvhiv0e3
JC5WB3Q2/t2dRXhKNOSxQJd5mBKGhHgKptTjzmnW8HF0uxCdy7XpKZ6vS4Kt8Gk=
=2JQe
-----END PGP SIGNATURE-----

From knepley at gmail.com  Wed Sep 16 07:34:40 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Sep 2015 07:34:40 -0500
Subject: [petsc-users] block matrix in serial
In-Reply-To: <55F95E31.6010404@imperial.ac.uk>
References: <55F8CE54.5010707@auckland.ac.nz>
	<CAMYG4G=i2O7nFpaVdrqqyYoEVtzHAyRt-MQmRKcfE2c-MWoezQ@mail.gmail.com>
	<55F95E31.6010404@imperial.ac.uk>
Message-ID: <CAMYG4G=7A8i_ovHyuFYM004Ek8xeAucjdDXi=sG0Nc21DjzQpg@mail.gmail.com>

On Wed, Sep 16, 2015 at 7:18 AM, Lawrence Mitchell <
lawrence.mitchell at imperial.ac.uk> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 16/09/15 12:40, Matthew Knepley wrote:
> > On Tue, Sep 15, 2015 at 9:05 PM, Adrian Croucher
> > <a.croucher at auckland.ac.nz <mailto:a.croucher at auckland.ac.nz>>
> > wrote:
> >
> > hi
> >
> > I have a test code (attached) that sets up a finite volume mesh
> > using DMPlex, with 2 degrees of freedom per cell.
> >
> > I then create a matrix using DMCreateMatrix(), having used
> > DMSetMatType() to set the matrix type to MATBAIJ or MATMPIBAIJ, to
> > take advantage of the block structure.
> >
> > This works OK and gives me the expected matrix structure when I
> > run on > 1 processor, but neither MATBAIJ or MATMPIBAIJ works if I
> > run it in serial:
> >
> >
> > Plex should automatically discover the block size from the Section.
> > If not, it uses block size 1. I have to look at the example to see
> > why the discovery is not working. Do you have any constraints?
>
> It looks like block discovery in parallel effectively always
> determines a block size of 1.  Running with -mat_view ::ascii_info gives:
>
> Mat Object: 2 MPI processes
>   type: mpibaij
>   rows=20, cols=20
>   total: nonzeros=112, allocated nonzeros=112
>   total number of mallocs used during MatSetValues calls =0
>       block size is 1
>                     ^^^
>
> The block size discovery does this:
>
>         for (p = pStart; p < pEnd; ++p) {
>           ierr = PetscSectionGetDof(sectionGlobal, p, &dof);CHKERRQ(ierr);
>           ierr = PetscSectionGetConstraintDof(sectionGlobal, p,
> &cdof);CHKERRQ(ierr);
>           if (dof-cdof) {
>             if (bs < 0) {
>               bs = dof-cdof;
>             } else if (bs != dof-cdof) {
>               /* Layout does not admit a pointwise block size */
>               bs = 1;
>               break;
>             }
>           }
>         }
>
> In parallel, some of the dofs are remote (encoded as negative).  So we
> run through seeing (dof - cdof) == 2, until we hit a "remote" point at
> when we see (dof - cdof) != 2 and then we break out and set bs = 1.
>
> In serial, we correctly determine bs == 2.  But then
> DMGetLocalToGlobalMapping always does
>
>       ierr = ISLocalToGlobalMappingCreate(PETSC_COMM_SELF, 1,size,
> ltog, PETSC_OWN_POINTER, &dm->ltogmap);CHKERRQ(ierr);
>
>
> i.e. is set with block size == 1.
>
> So there are two bugs here.
>
> 1. In parallel, block size discovery in Mat creation has gone wrong
>

Crap. Hoist on my own petard. Okay I will fix this.


> 2. (Always), the lgmap has block size of 1, irrespective of the
> discovered block size from the section.
>

Yep. This can also be fixed. It should work regardless, but would be better
with blocking.

  Thanks

     Matt


> Lawrence
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQEcBAEBAgAGBQJV+V4tAAoJECOc1kQ8PEYvqO4IAN4+oIgtBevvDAughPgUVOzq
> kESJkb0Bx4a7+y47IPsY/SOuOMjVgErz2SO3tGd7+K/U5fstojJbIC7zZqPIERn0
> S68lH3s+y1pVqmcIMFIorz6is+u46M2xIGS6MLe6d0DluslyThaaA7lMhuIvJkIX
> FC8QmtqCsIvHv10VuNll/81UJ3pXSZ+E81+Rs6pBhoGMCnRSXdMgFEtfWBBGL2JR
> byEOAueTmY+YZXJ5JINxsHG1C1lep5wyYfAERDiRaD9bhCsE4mf9z/yFdvhiv0e3
> JC5WB3Q2/t2dRXhKNOSxQJd5mBKGhHgKptTjzmnW8HF0uxCdy7XpKZ6vS4Kt8Gk=
> =2JQe
> -----END PGP SIGNATURE-----
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150916/0099906c/attachment.html>

From adlinds3 at ncsu.edu  Wed Sep 16 08:28:50 2015
From: adlinds3 at ncsu.edu (Alexander Lindsay)
Date: Wed, 16 Sep 2015 09:28:50 -0400
Subject: [petsc-users] DIVERGED_FNORM_NAN
Message-ID: <55F96E92.1090608@ncsu.edu>

Hi all,

I've built a MOOSE application and the problem I'm studying solves great 
using serial LU. However, when I try to solve in parallel using super 
LU, I encounter many DIVERGED_LINE_SEARCH errors using the default bt 
line search. If I switch to line_search=none, then instead of 
DIVERGED_LINE_SEARCH errors I get DIVERGED_FNORM_NAN errors. I do not 
have a lot of debugging experience, but I am slowly gaining more. Is 
there a good way to get more information about what's causing the NAN's? 
I.e. a specific residual or jacobian statement?

This may not even be an appropriate list to post this problem in (could 
be more on the MOOSE side); apologies if that's the case.

Alex

From knepley at gmail.com  Wed Sep 16 09:58:45 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Sep 2015 09:58:45 -0500
Subject: [petsc-users] DIVERGED_FNORM_NAN
In-Reply-To: <55F96E92.1090608@ncsu.edu>
References: <55F96E92.1090608@ncsu.edu>
Message-ID: <CAMYG4Gkj7Ki5frtL+ZNGMeESY7kunw3-n+KJQWwLAKwrmutGHw@mail.gmail.com>

On Wed, Sep 16, 2015 at 8:28 AM, Alexander Lindsay <adlinds3 at ncsu.edu>
wrote:

> Hi all,
>
> I've built a MOOSE application and the problem I'm studying solves great
> using serial LU. However, when I try to solve in parallel using super LU, I
> encounter many DIVERGED_LINE_SEARCH errors using the default bt line
> search. If I switch to line_search=none, then instead of
> DIVERGED_LINE_SEARCH errors I get DIVERGED_FNORM_NAN errors. I do not have
> a lot of debugging experience, but I am slowly gaining more. Is there a
> good way to get more information about what's causing the NAN's? I.e. a
> specific residual or jacobian statement?
>
> This may not even be an appropriate list to post this problem in (could be
> more on the MOOSE side); apologies if that's the case.
>

The NaN means something has gone wrong. However, there are a few
SuperLU_dist bugs that we know of. You could try:

  - Upgrading PETSc to the latest patch, which should download the fixed
SuperLU_dist

  - Trying MUMPS instead

  Thanks,

     Matt


> Alex
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150916/f578ed70/attachment.html>

From bikash at umich.edu  Wed Sep 16 14:08:33 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Wed, 16 Sep 2015 15:08:33 -0400
Subject: [petsc-users] MatMatMult vs series of MatMult
Message-ID: <CA+H+vqJAbecSnMgtMPEp0WnPqVF5tVtFpdAKu7HuWgTV4+M=sw@mail.gmail.com>

Hi,

I have sparse matrix of MATMPIAIJ format that needs to be multiplied with a
series of vectors and the resultant set of vectors are to be stored into a
matrix. I can do that in two ways:

1. Perform series of MatMult operations and store the resultant vectors in
a MATMPIDENSE matrix; or
2. Store the vectors in a MATMPIDENSE matrix and perform a MatMatMult
operation.

I want to know which of these two approaches will be more efficient and if
there is a crossover in terms of the number of vectors for which one
approach starts performing better than the other.

Thanks,
Bikash

-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150916/0d1276b5/attachment.html>

From xzhao99 at gmail.com  Wed Sep 16 14:09:56 2015
From: xzhao99 at gmail.com (Xujun Zhao)
Date: Wed, 16 Sep 2015 14:09:56 -0500
Subject: [petsc-users] EPS iteration number
Message-ID: <CAHOKZ67u2e3sKvXhN5MWXPTD4qWMi1ZZ5WSuosJSx04YTafTJg@mail.gmail.com>

Hi all,

I am solving a max/min eigenvalue problem, and call EPSGetIterationNumber()
to get the iteration number for convergence. However, it always returns 1.
This is a little confused, because I can see my subroutine of MATOP_MULT
operation has been called 6 or 16 times with different configurations.

The ConvergedReason returns 2, so it looks like the eigenvalue is correct.
also, it shows
 1 EPS nconv=2 first unconverged value (error) 0.0158234 (2.15550644e-03)

what does it mean?

Thanks.

Xujun



/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

   Solve the eigensystem and get the solution

   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

  ierr = PetscPrintf(PETSC_COMM_WORLD,"EPS solve starts ...\n");

  ierr = EPSSolve(eps);CHKERRQ(ierr);

  ierr = PetscPrintf(PETSC_COMM_WORLD,"\n--->test: EPS solve results:\n");

  ierr = EPSGetEigenvalue(eps,0,&value,NULL); CHKERRQ(ierr);

  ierr = PetscPrintf(PETSC_COMM_WORLD,"          the eigenvalue:
%f\n",value);

  ierr = EPSGetType(eps,&type);CHKERRQ(ierr);

  ierr = PetscPrintf(PETSC_COMM_WORLD,"          Solution method:
%s\n",type);

  ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);

  ierr = PetscPrintf(PETSC_COMM_WORLD,"          Number of iterations of
the method: %D\n",its);

  ierr = EPSGetConvergedReason(eps,&reason);

  ierr = PetscPrintf(PETSC_COMM_WORLD,"          EPS converged reason :
%D\n\n",reason);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150916/3bd887c9/attachment.html>

From jroman at dsic.upv.es  Wed Sep 16 14:32:58 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 16 Sep 2015 21:32:58 +0200
Subject: [petsc-users] EPS iteration number
In-Reply-To: <CAHOKZ67u2e3sKvXhN5MWXPTD4qWMi1ZZ5WSuosJSx04YTafTJg@mail.gmail.com>
References: <CAHOKZ67u2e3sKvXhN5MWXPTD4qWMi1ZZ5WSuosJSx04YTafTJg@mail.gmail.com>
Message-ID: <7C165876-A01F-493D-9AB4-921E28E5752F@dsic.upv.es>


> El 16/9/2015, a las 21:09, Xujun Zhao <xzhao99 at gmail.com> escribi?:
> 
> Hi all,
> 
> I am solving a max/min eigenvalue problem, and call EPSGetIterationNumber() to get the iteration number for convergence. However, it always returns 1. This is a little confused, because I can see my subroutine of MATOP_MULT operation has been called 6 or 16 times with different configurations.
> 
> The ConvergedReason returns 2, so it looks like the eigenvalue is correct. 
> also, it shows
>  1 EPS nconv=2 first unconverged value (error) 0.0158234 (2.15550644e-03)
> 
> what does it mean?
> 
> Thanks.
> 
> Xujun

In EPS the iteration number refers to ?outer? iterations, that is, restarts. So in Krylov-Schur with ncv=16 the first outer iteration will perform 16 MatMult?s and in subsequent outer iterations it will perform 8 MatMult?s.

Jose


From xzhao99 at gmail.com  Wed Sep 16 14:44:58 2015
From: xzhao99 at gmail.com (Xujun Zhao)
Date: Wed, 16 Sep 2015 14:44:58 -0500
Subject: [petsc-users] EPS iteration number
In-Reply-To: <7C165876-A01F-493D-9AB4-921E28E5752F@dsic.upv.es>
References: <CAHOKZ67u2e3sKvXhN5MWXPTD4qWMi1ZZ5WSuosJSx04YTafTJg@mail.gmail.com>
	<7C165876-A01F-493D-9AB4-921E28E5752F@dsic.upv.es>
Message-ID: <CAHOKZ67f4n+6RZ0DkMqcdDA1=V2sg98=Ab9BM-kUngXKu9psqg@mail.gmail.com>

The following is another test, which shows the outer iteration is 2, and
MatMult is performed 24 times. So it means the first outer iteration
performs 16, and the subsequent iterations performs 8. (24=16+8)

What parameters can I tune to accelerate the convergence? Will changing ncv
work better?
The following nconv=1 and unconverged value, what do they mean?

2 EPS nconv=1 first unconverged value (error) 0.0140231 (1.32666590e-02)

--->test: EPS solve results:
          the eigenvalue: 0.009301
          Solution method: krylovschur
          Number of iterations of the method: 2
          EPS converged reason : 2

On Wed, Sep 16, 2015 at 2:32 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> > El 16/9/2015, a las 21:09, Xujun Zhao <xzhao99 at gmail.com> escribi?:
> >
> > Hi all,
> >
> > I am solving a max/min eigenvalue problem, and call
> EPSGetIterationNumber() to get the iteration number for convergence.
> However, it always returns 1. This is a little confused, because I can see
> my subroutine of MATOP_MULT operation has been called 6 or 16 times with
> different configurations.
> >
> > The ConvergedReason returns 2, so it looks like the eigenvalue is
> correct.
> > also, it shows
> >  1 EPS nconv=2 first unconverged value (error) 0.0158234 (2.15550644e-03)
> >
> > what does it mean?
> >
> > Thanks.
> >
> > Xujun
>
> In EPS the iteration number refers to ?outer? iterations, that is,
> restarts. So in Krylov-Schur with ncv=16 the first outer iteration will
> perform 16 MatMult?s and in subsequent outer iterations it will perform 8
> MatMult?s.
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150916/c9cfecd5/attachment.html>

From hzhang at mcs.anl.gov  Wed Sep 16 15:05:27 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Wed, 16 Sep 2015 15:05:27 -0500
Subject: [petsc-users] MatMatMult vs series of MatMult
In-Reply-To: <CA+H+vqJAbecSnMgtMPEp0WnPqVF5tVtFpdAKu7HuWgTV4+M=sw@mail.gmail.com>
References: <CA+H+vqJAbecSnMgtMPEp0WnPqVF5tVtFpdAKu7HuWgTV4+M=sw@mail.gmail.com>
Message-ID: <CAGCphBvhwBzTjBM96-gZOSYb+wq1mZyvPZrGgK8hMCycJskLRg@mail.gmail.com>

Bikash:
>
>
> I have sparse matrix of MATMPIAIJ format that needs to be multiplied with
> a series of vectors and the resultant set of vectors are to be stored into
> a matrix. I can do that in two ways:
>
> 1. Perform series of MatMult operations and store the resultant vectors in
> a MATMPIDENSE matrix; or
> 2. Store the vectors in a MATMPIDENSE matrix and perform a MatMatMult
> operation.
>

2 would be more efficient.

Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150916/bfde3774/attachment.html>

From bikash at umich.edu  Wed Sep 16 15:32:16 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Wed, 16 Sep 2015 16:32:16 -0400
Subject: [petsc-users] MatMatMult vs series of MatMult
In-Reply-To: <CAGCphBvhwBzTjBM96-gZOSYb+wq1mZyvPZrGgK8hMCycJskLRg@mail.gmail.com>
References: <CA+H+vqJAbecSnMgtMPEp0WnPqVF5tVtFpdAKu7HuWgTV4+M=sw@mail.gmail.com>
	<CAGCphBvhwBzTjBM96-gZOSYb+wq1mZyvPZrGgK8hMCycJskLRg@mail.gmail.com>
Message-ID: <CA+H+vqKUB21X1YsXt17uvTgtW2+TkKs=LazZviQfYm1qSd=k0g@mail.gmail.com>

Thanks a lot, Hong.

On Wed, Sep 16, 2015 at 4:05 PM, Hong <hzhang at mcs.anl.gov> wrote:

> Bikash:
>>
>>
>> I have sparse matrix of MATMPIAIJ format that needs to be multiplied with
>> a series of vectors and the resultant set of vectors are to be stored into
>> a matrix. I can do that in two ways:
>>
>> 1. Perform series of MatMult operations and store the resultant vectors
>> in a MATMPIDENSE matrix; or
>> 2. Store the vectors in a MATMPIDENSE matrix and perform a MatMatMult
>> operation.
>>
>
> 2 would be more efficient.
>
> Hong
>



-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150916/3226e085/attachment.html>

From jroman at dsic.upv.es  Wed Sep 16 16:18:37 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 16 Sep 2015 23:18:37 +0200
Subject: [petsc-users] EPS iteration number
In-Reply-To: <CAHOKZ67f4n+6RZ0DkMqcdDA1=V2sg98=Ab9BM-kUngXKu9psqg@mail.gmail.com>
References: <CAHOKZ67u2e3sKvXhN5MWXPTD4qWMi1ZZ5WSuosJSx04YTafTJg@mail.gmail.com>
	<7C165876-A01F-493D-9AB4-921E28E5752F@dsic.upv.es>
	<CAHOKZ67f4n+6RZ0DkMqcdDA1=V2sg98=Ab9BM-kUngXKu9psqg@mail.gmail.com>
Message-ID: <709CE914-ABBE-45BA-81C5-18CB784E94BF@dsic.upv.es>


> El 16/9/2015, a las 21:44, Xujun Zhao <xzhao99 at gmail.com> escribi?:
> 
> The following is another test, which shows the outer iteration is 2, and MatMult is performed 24 times. So it means the first outer iteration performs 16, and the subsequent iterations performs 8. (24=16+8)
> 
> What parameters can I tune to accelerate the convergence? Will changing ncv work better?
> The following nconv=1 and unconverged value, what do they mean?
> 
> 2 EPS nconv=1 first unconverged value (error) 0.0140231 (1.32666590e-02)
> 
> --->test: EPS solve results:
>           the eigenvalue: 0.009301
>           Solution method: krylovschur
>           Number of iterations of the method: 2
>           EPS converged reason : 2

2 outer iterations is very fast, your problem seems very easy. You can try changing ncv but I would bet you won?t get the result with much less than 24 MatMult?s. Just try.

The monitor you are using shows the number of converged eigenvalues and the first non-converged Ritz value.

Jose


From bsmith at mcs.anl.gov  Wed Sep 16 17:18:00 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 16 Sep 2015 17:18:00 -0500
Subject: [petsc-users] DIVERGED_FNORM_NAN
In-Reply-To: <55F96E92.1090608@ncsu.edu>
References: <55F96E92.1090608@ncsu.edu>
Message-ID: <B870B2D8-998A-4B9B-9F59-936EFB9173F6@mcs.anl.gov>


> On Sep 16, 2015, at 8:28 AM, Alexander Lindsay <adlinds3 at ncsu.edu> wrote:
> 
> Hi all,
> 
> I've built a MOOSE application and the problem I'm studying solves great using serial LU. However, when I try to solve in parallel using super LU, I encounter many DIVERGED_LINE_SEARCH errors using the default bt line search. If I switch to line_search=none, then instead of DIVERGED_LINE_SEARCH errors I get DIVERGED_FNORM_NAN errors. I do not have a lot of debugging experience, but I am slowly gaining more. Is there a good way to get more information about what's causing the NAN's? I.e. a specific residual or jacobian statement?
> 

   It would be useful to have full output with -snes_monitor -snes_converged_reason -snes_linesearch_monitor -ksp_monitor_true_residual -ksp_converged_reason

   The original DIVERGED_LINE_SEARCH is almost always an indication that the step direction is not a Newton step. This needs to be debugged see http://www.mcs.anl.gov/petsc/documentation/faq.html#newton   Switching to no line search is not a useful thing to do here.

  Barry


> This may not even be an appropriate list to post this problem in (could be more on the MOOSE side); apologies if that's the case.
> 
> Alex


From erand at mek.dtu.dk  Thu Sep 17 07:07:56 2015
From: erand at mek.dtu.dk (Erik Andreassen)
Date: Thu, 17 Sep 2015 12:07:56 +0000
Subject: [petsc-users] Preconditioner question ASM vs SOR
Message-ID: <300719361A150149BDBDF898D5BF89CE063D5726@ait-pex01mbx02.win.dtu.dk>

I have played a bit with preconditioners for the different levels in PCMG, and among others I have tried additive Schwartz (ASM), which does not seem to work very well - especially with no overlap. I'm curious to hear if anyone can explain why ASM with no overlap performs so much worse than SOR?

Thanks,
Erik

PS: I had a previous question about setting levels in GAMG, and my first impression is (in line with what Mark Adams wrote) that it is better to leave it to figure it out automatically. Also when using it as the coarse level preconditioner in a PCMG.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/bbf0975b/attachment.html>

From knepley at gmail.com  Thu Sep 17 07:11:30 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Sep 2015 07:11:30 -0500
Subject: [petsc-users] Preconditioner question ASM vs SOR
In-Reply-To: <300719361A150149BDBDF898D5BF89CE063D5726@ait-pex01mbx02.win.dtu.dk>
References: <300719361A150149BDBDF898D5BF89CE063D5726@ait-pex01mbx02.win.dtu.dk>
Message-ID: <CAMYG4Gms9briKhCiJGqL=qZ6XRS0mFYFzHaGRuUWYVFu43iA6A@mail.gmail.com>

On Thu, Sep 17, 2015 at 7:07 AM, Erik Andreassen <erand at mek.dtu.dk> wrote:

> I have played a bit with preconditioners for the different levels in PCMG,
> and among others I have tried additive Schwartz (ASM), which does not seem
> to work very well ? especially with no overlap. I?m curious to hear if
> anyone can explain why ASM with no overlap performs so much worse than SOR?
>

The point of a smoother is to wipe out the high frequencies in the error.
SOR does this (provably for the Laplacian), but there is no
reason ASM/ILU should do this. If this does not happen, then these parts of
the solution remain uncorrected when you go to the
coarse grid, and the convergence breaks down.

   Matt


> Thanks,
>
> Erik
>
>
>
> PS: I had a previous question about setting levels in GAMG, and my first
> impression is (in line with what Mark Adams wrote) that it is better to
> leave it to figure it out automatically. Also when using it as the coarse
> level preconditioner in a PCMG.
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/fa63f910/attachment.html>

From erand at mek.dtu.dk  Thu Sep 17 07:39:08 2015
From: erand at mek.dtu.dk (Erik Andreassen)
Date: Thu, 17 Sep 2015 12:39:08 +0000
Subject: [petsc-users] Preconditioner question ASM vs SOR
In-Reply-To: <CAMYG4Gms9briKhCiJGqL=qZ6XRS0mFYFzHaGRuUWYVFu43iA6A@mail.gmail.com>
References: <300719361A150149BDBDF898D5BF89CE063D5726@ait-pex01mbx02.win.dtu.dk>
	<CAMYG4Gms9briKhCiJGqL=qZ6XRS0mFYFzHaGRuUWYVFu43iA6A@mail.gmail.com>
Message-ID: <300719361A150149BDBDF898D5BF89CE063D5749@ait-pex01mbx02.win.dtu.dk>

I think I understand; does this mean that it is possible to change the sub PC type to SOR? Or is there a specific reason why the sub type is ILU?

The reason why I am asking, is that when I look at the residuals coming out of PCMG, they seem to be largest on local domain boundaries (corresponding to the partitions). Therefore, I?d like to use ASM with overlap instead of SOR as the preconditioner on the coarser levels (to smooth these errors out), but with the ASM/ILU combination this does not work at all (in accordance with your comment on high frequencies).

Thanks again,
Erik

From: Matthew Knepley [mailto:knepley at gmail.com]
Sent: 17. september 2015 14:12
To: Erik Andreassen
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Preconditioner question ASM vs SOR

On Thu, Sep 17, 2015 at 7:07 AM, Erik Andreassen <erand at mek.dtu.dk<mailto:erand at mek.dtu.dk>> wrote:
I have played a bit with preconditioners for the different levels in PCMG, and among others I have tried additive Schwartz (ASM), which does not seem to work very well ? especially with no overlap. I?m curious to hear if anyone can explain why ASM with no overlap performs so much worse than SOR?

The point of a smoother is to wipe out the high frequencies in the error. SOR does this (provably for the Laplacian), but there is no
reason ASM/ILU should do this. If this does not happen, then these parts of the solution remain uncorrected when you go to the
coarse grid, and the convergence breaks down.

   Matt

Thanks,
Erik

PS: I had a previous question about setting levels in GAMG, and my first impression is (in line with what Mark Adams wrote) that it is better to leave it to figure it out automatically. Also when using it as the coarse level preconditioner in a PCMG.



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/bf20630d/attachment.html>

From knepley at gmail.com  Thu Sep 17 07:58:01 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Sep 2015 07:58:01 -0500
Subject: [petsc-users] Preconditioner question ASM vs SOR
In-Reply-To: <300719361A150149BDBDF898D5BF89CE063D5749@ait-pex01mbx02.win.dtu.dk>
References: <300719361A150149BDBDF898D5BF89CE063D5726@ait-pex01mbx02.win.dtu.dk>
	<CAMYG4Gms9briKhCiJGqL=qZ6XRS0mFYFzHaGRuUWYVFu43iA6A@mail.gmail.com>
	<300719361A150149BDBDF898D5BF89CE063D5749@ait-pex01mbx02.win.dtu.dk>
Message-ID: <CAMYG4GkSd+XMada+swRPWC-sUN-RkjM8PX72Zk_WwTkvgMv_mQ@mail.gmail.com>

On Thu, Sep 17, 2015 at 7:39 AM, Erik Andreassen <erand at mek.dtu.dk> wrote:

> I think I understand; does this mean that it is possible to change the sub
> PC type to SOR? Or is there a specific reason why the sub type is ILU?
>

Yes, this is exactly it. We have a default of ILU because it works better
in a general purpose problem, but it usually not the solver you want to use
for any given problem.

  Thanks,

     Matt


> The reason why I am asking, is that when I look at the residuals coming
> out of PCMG, they seem to be largest on local domain boundaries
> (corresponding to the partitions). Therefore, I?d like to use ASM with
> overlap instead of SOR as the preconditioner on the coarser levels (to
> smooth these errors out), but with the ASM/ILU combination this does not
> work at all (in accordance with your comment on high frequencies).
>
>
>
> Thanks again,
>
> Erik
>
>
>
> *From:* Matthew Knepley [mailto:knepley at gmail.com]
> *Sent:* 17. september 2015 14:12
> *To:* Erik Andreassen
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] Preconditioner question ASM vs SOR
>
>
>
> On Thu, Sep 17, 2015 at 7:07 AM, Erik Andreassen <erand at mek.dtu.dk> wrote:
>
> I have played a bit with preconditioners for the different levels in PCMG,
> and among others I have tried additive Schwartz (ASM), which does not seem
> to work very well ? especially with no overlap. I?m curious to hear if
> anyone can explain why ASM with no overlap performs so much worse than SOR?
>
>
>
> The point of a smoother is to wipe out the high frequencies in the error.
> SOR does this (provably for the Laplacian), but there is no
>
> reason ASM/ILU should do this. If this does not happen, then these parts
> of the solution remain uncorrected when you go to the
>
> coarse grid, and the convergence breaks down.
>
>
>
>    Matt
>
>
>
> Thanks,
>
> Erik
>
>
>
> PS: I had a previous question about setting levels in GAMG, and my first
> impression is (in line with what Mark Adams wrote) that it is better to
> leave it to figure it out automatically. Also when using it as the coarse
> level preconditioner in a PCMG.
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/c63c753e/attachment-0001.html>

From adlinds3 at ncsu.edu  Thu Sep 17 09:24:51 2015
From: adlinds3 at ncsu.edu (Alexander Lindsay)
Date: Thu, 17 Sep 2015 10:24:51 -0400
Subject: [petsc-users] DIVERGED_FNORM_NAN
In-Reply-To: <CAMYG4Gkj7Ki5frtL+ZNGMeESY7kunw3-n+KJQWwLAKwrmutGHw@mail.gmail.com>
References: <55F96E92.1090608@ncsu.edu>
	<CAMYG4Gkj7Ki5frtL+ZNGMeESY7kunw3-n+KJQWwLAKwrmutGHw@mail.gmail.com>
Message-ID: <55FACD33.2020604@ncsu.edu>

It solved beautifully with MUMPS. Thank you Matt!

Alex

On 09/16/2015 10:58 AM, Matthew Knepley wrote:
> On Wed, Sep 16, 2015 at 8:28 AM, Alexander Lindsay <adlinds3 at ncsu.edu 
> <mailto:adlinds3 at ncsu.edu>> wrote:
>
>     Hi all,
>
>     I've built a MOOSE application and the problem I'm studying solves
>     great using serial LU. However, when I try to solve in parallel
>     using super LU, I encounter many DIVERGED_LINE_SEARCH errors using
>     the default bt line search. If I switch to line_search=none, then
>     instead of DIVERGED_LINE_SEARCH errors I get DIVERGED_FNORM_NAN
>     errors. I do not have a lot of debugging experience, but I am
>     slowly gaining more. Is there a good way to get more information
>     about what's causing the NAN's? I.e. a specific residual or
>     jacobian statement?
>
>     This may not even be an appropriate list to post this problem in
>     (could be more on the MOOSE side); apologies if that's the case.
>
>
> The NaN means something has gone wrong. However, there are a few 
> SuperLU_dist bugs that we know of. You could try:
>
>   - Upgrading PETSc to the latest patch, which should download the 
> fixed SuperLU_dist
>
>   - Trying MUMPS instead
>
>   Thanks,
>
>      Matt
>
>     Alex
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/1eefffd5/attachment.html>

From xzhao99 at gmail.com  Thu Sep 17 14:33:35 2015
From: xzhao99 at gmail.com (Xujun Zhao)
Date: Thu, 17 Sep 2015 14:33:35 -0500
Subject: [petsc-users] EPS iteration number
In-Reply-To: <709CE914-ABBE-45BA-81C5-18CB784E94BF@dsic.upv.es>
References: <CAHOKZ67u2e3sKvXhN5MWXPTD4qWMi1ZZ5WSuosJSx04YTafTJg@mail.gmail.com>
	<7C165876-A01F-493D-9AB4-921E28E5752F@dsic.upv.es>
	<CAHOKZ67f4n+6RZ0DkMqcdDA1=V2sg98=Ab9BM-kUngXKu9psqg@mail.gmail.com>
	<709CE914-ABBE-45BA-81C5-18CB784E94BF@dsic.upv.es>
Message-ID: <CAHOKZ65ev9onAY+DnZPZTu1Nebcot_-nyxXum7AYHX2L4WJaxA@mail.gmail.com>

Hi Jose,

This is only a very simple test. 2 outer iterations perform 24 MatMult
operation, which means I have to solve a Finite element system with
millions of DOFs for 24 times. This is only for a polymer system with two
beads. For a polymer chain with hundreds of beads and springs, I need more
outer iterations to get its max/min eigenvalues, which means I have to
perform more computations of the finite element system. Those solves happen
only for one time step. When polymer beads move, I need to redo everything.

In other words, the eigenvalues (max and min) have to be solved every time
step or every several time steps. Therefore, it is very expensive! My
question is that if It is possible to use the eigenvalues obtained from
last time step as an initial guess to accelerate the convergence. Thank you
for your help.

Xujun





On Wed, Sep 16, 2015 at 4:18 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> > El 16/9/2015, a las 21:44, Xujun Zhao <xzhao99 at gmail.com> escribi?:
> >
> > The following is another test, which shows the outer iteration is 2, and
> MatMult is performed 24 times. So it means the first outer iteration
> performs 16, and the subsequent iterations performs 8. (24=16+8)
> >
> > What parameters can I tune to accelerate the convergence? Will changing
> ncv work better?
> > The following nconv=1 and unconverged value, what do they mean?
> >
> > 2 EPS nconv=1 first unconverged value (error) 0.0140231 (1.32666590e-02)
> >
> > --->test: EPS solve results:
> >           the eigenvalue: 0.009301
> >           Solution method: krylovschur
> >           Number of iterations of the method: 2
> >           EPS converged reason : 2
>
> 2 outer iterations is very fast, your problem seems very easy. You can try
> changing ncv but I would bet you won?t get the result with much less than
> 24 MatMult?s. Just try.
>
> The monitor you are using shows the number of converged eigenvalues and
> the first non-converged Ritz value.
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/fe878dd4/attachment.html>

From jychang48 at gmail.com  Thu Sep 17 15:55:00 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Thu, 17 Sep 2015 14:55:00 -0600
Subject: [petsc-users] TaoSetVariableBounds in petsc4py
Message-ID: <CAP2=TMg+SOajPw1_9QTcYYtTBHRpGFqbbOwT7dwsBAPdsnOEcg@mail.gmail.com>

Hi all,

I am attempting to use TaoSetVariableBounds via petsc4py. Is there a way to
set only lower bounds or upper bounds and not both? In PETSc I could
replace one of the Vecs with NULL, but how would I do that with
tao.setVariableBounds(...)?

It seems right now I can only do tao.setVariableBounds(lb_vec,ub_vec), and
from inspection of the python source code, it seems I have to specify both?

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/f2e49e56/attachment.html>

From knepley at gmail.com  Thu Sep 17 15:59:06 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Sep 2015 15:59:06 -0500
Subject: [petsc-users] TaoSetVariableBounds in petsc4py
In-Reply-To: <CAP2=TMg+SOajPw1_9QTcYYtTBHRpGFqbbOwT7dwsBAPdsnOEcg@mail.gmail.com>
References: <CAP2=TMg+SOajPw1_9QTcYYtTBHRpGFqbbOwT7dwsBAPdsnOEcg@mail.gmail.com>
Message-ID: <CAMYG4GkZXsXv1rU33nPxBdY0mU8osD=0vBNMHKaom42+n3kRGA@mail.gmail.com>

On Thu, Sep 17, 2015 at 3:55 PM, Justin Chang <jychang48 at gmail.com> wrote:

> Hi all,
>
> I am attempting to use TaoSetVariableBounds via petsc4py. Is there a way
> to set only lower bounds or upper bounds and not both? In PETSc I could
> replace one of the Vecs with NULL, but how would I do that with
> tao.setVariableBounds(...)?
>
> It seems right now I can only do tao.setVariableBounds(lb_vec,ub_vec), and
> from inspection of the python source code, it seems I have to specify both?
>

Can you jsut give None?

  Matt


> Thanks,
> Justin
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/30426051/attachment.html>

From jychang48 at gmail.com  Thu Sep 17 16:00:47 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Thu, 17 Sep 2015 15:00:47 -0600
Subject: [petsc-users] TaoSetVariableBounds in petsc4py
In-Reply-To: <CAMYG4GkZXsXv1rU33nPxBdY0mU8osD=0vBNMHKaom42+n3kRGA@mail.gmail.com>
References: <CAP2=TMg+SOajPw1_9QTcYYtTBHRpGFqbbOwT7dwsBAPdsnOEcg@mail.gmail.com>
	<CAMYG4GkZXsXv1rU33nPxBdY0mU8osD=0vBNMHKaom42+n3kRGA@mail.gmail.com>
Message-ID: <CAP2=TMhYyhSM+S4qqNEL7r5bGPTy1vh6e6sX4yYjghG2BmnkOw@mail.gmail.com>

Matt,

This is the error I get if I use None:

Traceback (most recent call last):
  File "2D_plume_ADR_ex1.py", line 146, in <module>
    tao_D.setVariableBounds(lb_vec,None)
  File "PETSc/TAO.pyx", line 198, in petsc4py.PETSc.TAO.setVariableBounds
(src/petsc4py.PETSc.c:189484)
TypeError: Cannot convert NoneType to petsc4py.PETSc.Vec

Thanks,
Justin

On Thu, Sep 17, 2015 at 2:59 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Sep 17, 2015 at 3:55 PM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> Hi all,
>>
>> I am attempting to use TaoSetVariableBounds via petsc4py. Is there a way
>> to set only lower bounds or upper bounds and not both? In PETSc I could
>> replace one of the Vecs with NULL, but how would I do that with
>> tao.setVariableBounds(...)?
>>
>> It seems right now I can only do tao.setVariableBounds(lb_vec,ub_vec),
>> and from inspection of the python source code, it seems I have to specify
>> both?
>>
>
> Can you jsut give None?
>
>   Matt
>
>
>> Thanks,
>> Justin
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/8cc8b4e8/attachment.html>

From jason.sarich at gmail.com  Thu Sep 17 16:06:48 2015
From: jason.sarich at gmail.com (Jason Sarich)
Date: Thu, 17 Sep 2015 16:06:48 -0500
Subject: [petsc-users] TaoSetVariableBounds in petsc4py
In-Reply-To: <4b5a535121f24b2e8f37745b9e9c1bde@GEORGE.anl.gov>
References: <CAP2=TMg+SOajPw1_9QTcYYtTBHRpGFqbbOwT7dwsBAPdsnOEcg@mail.gmail.com>
	<CAMYG4GkZXsXv1rU33nPxBdY0mU8osD=0vBNMHKaom42+n3kRGA@mail.gmail.com>
	<4b5a535121f24b2e8f37745b9e9c1bde@GEORGE.anl.gov>
Message-ID: <CAFcybM=VWf9Y1+-kJaaCDfhgUAb0E4hoUtwqBuM9h-35o+nOvA@mail.gmail.com>

Hi Justin,

The C interface allows null to be passed as one of the vectors, this looks
like an issue with the petsc4py interface specifically.

You can create the unneeded bound vector and set it to infinity or negative
infinity, and pass it to this function instead of using none (TAO will do
this anyway if you pass in NULL).

Jason


On Thu, Sep 17, 2015 at 4:00 PM, Justin Chang <jychang48 at gmail.com> wrote:

> Matt,
>
> This is the error I get if I use None:
>
> Traceback (most recent call last):
>   File "2D_plume_ADR_ex1.py", line 146, in <module>
>     tao_D.setVariableBounds(lb_vec,None)
>   File "PETSc/TAO.pyx", line 198, in petsc4py.PETSc.TAO.setVariableBounds
> (src/petsc4py.PETSc.c:189484)
> TypeError: Cannot convert NoneType to petsc4py.PETSc.Vec
>
> Thanks,
> Justin
>
> On Thu, Sep 17, 2015 at 2:59 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Thu, Sep 17, 2015 at 3:55 PM, Justin Chang <jychang48 at gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I am attempting to use TaoSetVariableBounds via petsc4py. Is there a way
>>> to set only lower bounds or upper bounds and not both? In PETSc I could
>>> replace one of the Vecs with NULL, but how would I do that with
>>> tao.setVariableBounds(...)?
>>>
>>> It seems right now I can only do tao.setVariableBounds(lb_vec,ub_vec),
>>> and from inspection of the python source code, it seems I have to specify
>>> both?
>>>
>>
>> Can you jsut give None?
>>
>>   Matt
>>
>>
>>> Thanks,
>>> Justin
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/c453eb9a/attachment.html>

From jychang48 at gmail.com  Thu Sep 17 17:24:01 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Thu, 17 Sep 2015 16:24:01 -0600
Subject: [petsc-users] TaoSetVariableBounds in petsc4py
In-Reply-To: <CAFcybM=VWf9Y1+-kJaaCDfhgUAb0E4hoUtwqBuM9h-35o+nOvA@mail.gmail.com>
References: <CAP2=TMg+SOajPw1_9QTcYYtTBHRpGFqbbOwT7dwsBAPdsnOEcg@mail.gmail.com>
	<CAMYG4GkZXsXv1rU33nPxBdY0mU8osD=0vBNMHKaom42+n3kRGA@mail.gmail.com>
	<4b5a535121f24b2e8f37745b9e9c1bde@GEORGE.anl.gov>
	<CAFcybM=VWf9Y1+-kJaaCDfhgUAb0E4hoUtwqBuM9h-35o+nOvA@mail.gmail.com>
Message-ID: <CAP2=TMjMQDKfsg6Yeeawn8-7mnbkU1fc2uAgnRzCUGEA9LGU0A@mail.gmail.com>

Ah that's right, silly me. Yeah I didn't think about that for some reason.
Thanks

On Thu, Sep 17, 2015 at 3:06 PM, Jason Sarich <jason.sarich at gmail.com>
wrote:

> Hi Justin,
>
> The C interface allows null to be passed as one of the vectors, this looks
> like an issue with the petsc4py interface specifically.
>
> You can create the unneeded bound vector and set it to infinity or
> negative infinity, and pass it to this function instead of using none (TAO
> will do this anyway if you pass in NULL).
>
> Jason
>
>
> On Thu, Sep 17, 2015 at 4:00 PM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> Matt,
>>
>> This is the error I get if I use None:
>>
>> Traceback (most recent call last):
>>   File "2D_plume_ADR_ex1.py", line 146, in <module>
>>     tao_D.setVariableBounds(lb_vec,None)
>>   File "PETSc/TAO.pyx", line 198, in petsc4py.PETSc.TAO.setVariableBounds
>> (src/petsc4py.PETSc.c:189484)
>> TypeError: Cannot convert NoneType to petsc4py.PETSc.Vec
>>
>> Thanks,
>> Justin
>>
>> On Thu, Sep 17, 2015 at 2:59 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Thu, Sep 17, 2015 at 3:55 PM, Justin Chang <jychang48 at gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am attempting to use TaoSetVariableBounds via petsc4py. Is there a
>>>> way to set only lower bounds or upper bounds and not both? In PETSc I could
>>>> replace one of the Vecs with NULL, but how would I do that with
>>>> tao.setVariableBounds(...)?
>>>>
>>>> It seems right now I can only do tao.setVariableBounds(lb_vec,ub_vec),
>>>> and from inspection of the python source code, it seems I have to specify
>>>> both?
>>>>
>>>
>>> Can you jsut give None?
>>>
>>>   Matt
>>>
>>>
>>>> Thanks,
>>>> Justin
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/fec7fc8d/attachment-0001.html>

From pvsang002 at gmail.com  Thu Sep 17 19:12:56 2015
From: pvsang002 at gmail.com (Sang pham van)
Date: Fri, 18 Sep 2015 07:12:56 +0700
Subject: [petsc-users] Release object
Message-ID: <CAOx7dQkXS2pfG2Kpk4+duLmPggQGJq=KMLZ-HAY4sGfuHfw+iw@mail.gmail.com>

Hi,

I have the below code section, every time it runs through this section,
used memory increases very much and not decreases then. Could you please
let me know what object did I forgot to release in the for-loop?

Many thanks.

PVS


  int mu_x[20]={0,1,0,0,2,0,0,1,0,1,3,0,0,2,2,1,0,1,0,1};
  int mu_y[20]={0,0,1,0,0,2,0,1,1,0,0,3,0,1,0,2,2,0,1,1};
  int mu_z[20]={0,0,0,1,0,0,2,0,1,1,0,0,3,0,1,0,1,2,2,1};
  for (int ib=0;ib<isur->BIPoints.N;ib++)
  {
    Mat W,V;
    int Mm = isur->iSolidCellsNearBIPoint[ib+1] -
isur->iSolidCellsNearBIPoint[ib];
    PetscScalar val=0;
    MatCreateSeqAIJ(PETSC_COMM_SELF,Mm,Mm,Mm,NULL,&W);
    MatCreateSeqAIJ(PETSC_COMM_SELF,Mm,L3D,L3D,NULL,&V);
    double xm, ym, zm, dm;
    for (int i=0;i<Mm;i++)
    {
      xm =
Mesh->Cells[isur->SolidCellsNearBIPoint[isur->iSolidCellsNearBIPoint[ib]+i]].Cen.xyz[0]
- isur->BIPoints.x[ib];
      ym =
Mesh->Cells[isur->SolidCellsNearBIPoint[isur->iSolidCellsNearBIPoint[ib]+i]].Cen.xyz[1]
- isur->BIPoints.y[ib];
      zm =
Mesh->Cells[isur->SolidCellsNearBIPoint[isur->iSolidCellsNearBIPoint[ib]+i]].Cen.xyz[2]
- isur->BIPoints.z[ib];
      dm = sqrt(xm*xm+ym*ym+zm*zm);
      val = WeightFunc(dm,isur->R_solid[ib]);
      MatSetValue(W,i,i,val,INSERT_VALUES);
    }
    MatAssemblyBegin(W,MAT_FINAL_ASSEMBLY);
    MatAssemblyEnd(W,MAT_FINAL_ASSEMBLY);

    for (int i=0;i<Mm;i++)
    for (int j=0;j<L3D;j++)
    {
      xm =
Mesh->Cells[isur->SolidCellsNearBIPoint[isur->iSolidCellsNearBIPoint[ib]+i]].Cen.xyz[0]
- isur->BIPoints.x[ib];
      ym =
Mesh->Cells[isur->SolidCellsNearBIPoint[isur->iSolidCellsNearBIPoint[ib]+i]].Cen.xyz[1]
- isur->BIPoints.y[ib];
      zm =
Mesh->Cells[isur->SolidCellsNearBIPoint[isur->iSolidCellsNearBIPoint[ib]+i]].Cen.xyz[2]
- isur->BIPoints.z[ib];
      val = power(xm,mu_x[j])*power(ym,mu_y[j])*power(zm,mu_z[j]);
      MatSetValue(V,i,j,val,INSERT_VALUES);
    }
    MatAssemblyBegin(V,MAT_FINAL_ASSEMBLY);
    MatAssemblyEnd(V,MAT_FINAL_ASSEMBLY);
    //if(ib==13) MatView(V,PETSC_VIEWER_STDOUT_SELF);

    /// Solve least square problem:
    Mat Amat,W2,VtW2,VtW2V,invVtW2V, VtW2V1Vt,I,invVtW2V_b;

    MatDuplicate(W,MAT_DO_NOT_COPY_VALUES,&W2);
    MatDuplicate(W2,MAT_DO_NOT_COPY_VALUES,&VtW2);
    MatDuplicate(W2,MAT_DO_NOT_COPY_VALUES,&VtW2V);

    MatMatMult(W,W,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&W2);

    MatTransposeMatMult(V,W2,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&VtW2);

    MatMatMult(VtW2,V,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&VtW2V);

    MatCreateSeqDense(PETSC_COMM_SELF,L3D,L3D,NULL,&I);
    MatDuplicate(I,MAT_DO_NOT_COPY_VALUES,&invVtW2V);

    Vec ivec;
    VecCreateSeq(PETSC_COMM_SELF,L3D,&ivec);
    VecSet(ivec,1.0);
    MatDiagonalSet(I,ivec,INSERT_VALUES);

    IS rperm,cperm;
    MatFactorInfo info;
    MatGetOrdering(VtW2V,  MATORDERINGNATURAL,  &rperm,  &cperm);
    MatFactorInfoInitialize(&info);
    MatLUFactor(VtW2V, rperm, cperm, &info);

    MatMatSolve(VtW2V,I,invVtW2V);

    MatConvert(invVtW2V,MATSEQAIJ,MAT_INITIAL_MATRIX,&invVtW2V_b);

MatMatTransposeMult(invVtW2V_b,V,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&VtW2V1Vt);
    MatMatMult(VtW2V1Vt,W2,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&Amat);
    for (int i=0;i<L3D;i++)
    for (int j=0;j<Mm;j++)
      MatGetValues(Amat,1,&i,1,&j,&A[ib][i][j]);

    MatDestroy(&W2);
    MatDestroy(&Amat);
    MatDestroy(&VtW2);
    MatDestroy(&VtW2V);
    MatDestroy(&invVtW2V);
    MatDestroy(&invVtW2V_b);
    MatDestroy(&VtW2V1Vt);
    MatDestroy(&I);

    MatDestroy(&W);
    MatDestroy(&V);
    VecDestroy(&ivec);
  }
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150918/7aa2f9a9/attachment.html>

From gideon.simpson at gmail.com  Thu Sep 17 21:14:00 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Thu, 17 Sep 2015 22:14:00 -0400
Subject: [petsc-users] early termination option in grid sequencing
Message-ID: <707162E4-23AF-469B-B772-F975004556CA@gmail.com>

I think I may have asked this before, but, if while using grid sequencing in a snes, if there is a failure at a coarser grid, is there a way to get the whole sequence to terminate, rather than to try to solve at the next refinement?

-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150917/87f6bc9e/attachment.html>

From jroman at dsic.upv.es  Fri Sep 18 02:49:16 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 18 Sep 2015 09:49:16 +0200
Subject: [petsc-users] EPS iteration number
In-Reply-To: <CAHOKZ65ev9onAY+DnZPZTu1Nebcot_-nyxXum7AYHX2L4WJaxA@mail.gmail.com>
References: <CAHOKZ67u2e3sKvXhN5MWXPTD4qWMi1ZZ5WSuosJSx04YTafTJg@mail.gmail.com>
	<7C165876-A01F-493D-9AB4-921E28E5752F@dsic.upv.es>
	<CAHOKZ67f4n+6RZ0DkMqcdDA1=V2sg98=Ab9BM-kUngXKu9psqg@mail.gmail.com>
	<709CE914-ABBE-45BA-81C5-18CB784E94BF@dsic.upv.es>
	<CAHOKZ65ev9onAY+DnZPZTu1Nebcot_-nyxXum7AYHX2L4WJaxA@mail.gmail.com>
Message-ID: <3A4019A7-CFA6-4BC3-8992-17CDC4EBF704@dsic.upv.es>


> El 17/9/2015, a las 21:33, Xujun Zhao <xzhao99 at gmail.com> escribi?:
> 
> Hi Jose,
> 
> This is only a very simple test. 2 outer iterations perform 24 MatMult operation, which means I have to solve a Finite element system with millions of DOFs for 24 times. This is only for a polymer system with two beads. For a polymer chain with hundreds of beads and springs, I need more outer iterations to get its max/min eigenvalues, which means I have to perform more computations of the finite element system. Those solves happen only for one time step. When polymer beads move, I need to redo everything. 
> 
> In other words, the eigenvalues (max and min) have to be solved every time step or every several time steps. Therefore, it is very expensive! My question is that if It is possible to use the eigenvalues obtained from last time step as an initial guess to accelerate the convergence. Thank you for your help.
> 
> Xujun

Eigenvalues cannot be used as initial guess. You can try passing the previous eigenvector with EPSSetInitialSpace(), but there is no guarantee it is effective to improve convergence.

Jose


From jychang48 at gmail.com  Fri Sep 18 03:48:06 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Fri, 18 Sep 2015 02:48:06 -0600
Subject: [petsc-users] Hypre's BoomerAMG vs PETSc's GAMG?
Message-ID: <CAP2=TMhGpE8UkvpTuB7uuvcwRU=4Jhb+at=oyiaRgPyz7pkZ5A@mail.gmail.com>

Hi all,

So I am working with some nice symmetric and positive definite
discretization, and have been told that algebraic multi-grid methods are
best for solving these types of problems.

I see that there's GAMG and there's Hypre's BoomerAMG (and perhaps others
too?) What exactly is the difference between these two? Do they have very
different implementations under the hood? Does one have better scalability
over another? Or more importantly, when would I choose one over the other?

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150918/058b12c1/attachment.html>

From jed at jedbrown.org  Fri Sep 18 06:36:09 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 18 Sep 2015 05:36:09 -0600
Subject: [petsc-users] Hypre's BoomerAMG vs PETSc's GAMG?
In-Reply-To: <CAP2=TMhGpE8UkvpTuB7uuvcwRU=4Jhb+at=oyiaRgPyz7pkZ5A@mail.gmail.com>
References: <CAP2=TMhGpE8UkvpTuB7uuvcwRU=4Jhb+at=oyiaRgPyz7pkZ5A@mail.gmail.com>
Message-ID: <87zj0k6jti.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:
> I see that there's GAMG and there's Hypre's BoomerAMG (and perhaps others
> too?) 

ML

> What exactly is the difference between these two? Do they have very
> different implementations under the hood? Does one have better
> scalability over another? Or more importantly, when would I choose one
> over the other?

BoomerAMG is classical AMG, which is a different coarsening strategy
From GAMG and ML which use smoothed aggregation.  The math is different
and the implementation is different.  ML and GAMG have similar math and
there exist configurations that are equivalent.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150918/5b86860f/attachment.pgp>

From gideon.simpson at gmail.com  Fri Sep 18 22:42:02 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Fri, 18 Sep 2015 23:42:02 -0400
Subject: [petsc-users] gamg and zero pivots
Message-ID: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>

I?m getting a zero pivot in LU factorization error when I use the gamg preconditioner.  I know there is some issue with my matrix, as I get the same error if I use pc_type lu in serial, or if I use pc_type bjacobi in parallel, both of which are corrected by adding -pc_factor_shift_type NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t quite seem to figure out the analogous command to use for the gamg preconditioner.

-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150918/3dc17378/attachment.html>

From bsmith at mcs.anl.gov  Fri Sep 18 23:37:34 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 18 Sep 2015 23:37:34 -0500
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
Message-ID: <9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>


> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I?m getting a zero pivot in LU factorization error when I use the gamg preconditioner.  I know there is some issue with my matrix, as I get the same error if I use pc_type lu in serial, or if I use pc_type bjacobi in parallel, both of which are corrected by adding -pc_factor_shift_type NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t quite seem to figure out the analogous command to use for the gamg preconditioner.

  You need to figure out the prefix to the option that is added by gamg to set it. One way is to run with the gamg options and -help and grep for factor_shift.   

   The option is like something like -mg_coarse_pc_factor_shift_type nonzero or -mg_coarse_sub_pc_factor_shift_type 

  Barry

> 
> -gideon
> 


From bsmith at mcs.anl.gov  Sat Sep 19 16:33:23 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 19 Sep 2015 16:33:23 -0500
Subject: [petsc-users] early termination option in grid sequencing
In-Reply-To: <707162E4-23AF-469B-B772-F975004556CA@gmail.com>
References: <707162E4-23AF-469B-B772-F975004556CA@gmail.com>
Message-ID: <30E59A8A-6712-4F32-85A1-EB5AE843C5C5@mcs.anl.gov>


  Yeah the current code is kind of goofy. I have changed master and next branches so the SNESSolve() returns on the first grid it fails on rather than interpolating to the finer grids and trying again there (where it would generally be hopeless).

   Barry

> On Sep 17, 2015, at 9:14 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I think I may have asked this before, but, if while using grid sequencing in a snes, if there is a failure at a coarser grid, is there a way to get the whole sequence to terminate, rather than to try to solve at the next refinement?
> 
> -gideon
> 


From gideon.simpson at gmail.com  Sat Sep 19 18:10:43 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Sat, 19 Sep 2015 19:10:43 -0400
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
Message-ID: <88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>

It gives zero pivots on:

-ksp_type gmres -pc_type gamg -mg_coarse_sub_pc_factor_shift_type NONZERO

It has no problem with:

-ksp_type gmres -pc_type bjacobi -sub_pc_factor_shift_type NONZERO

-gideon

> On Sep 19, 2015, at 12:37 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> I?m getting a zero pivot in LU factorization error when I use the gamg preconditioner.  I know there is some issue with my matrix, as I get the same error if I use pc_type lu in serial, or if I use pc_type bjacobi in parallel, both of which are corrected by adding -pc_factor_shift_type NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t quite seem to figure out the analogous command to use for the gamg preconditioner.
> 
>  You need to figure out the prefix to the option that is added by gamg to set it. One way is to run with the gamg options and -help and grep for factor_shift.   
> 
>   The option is like something like -mg_coarse_pc_factor_shift_type nonzero or -mg_coarse_sub_pc_factor_shift_type 
> 
>  Barry
> 
>> 
>> -gideon
>> 
> 


From knepley at gmail.com  Sat Sep 19 18:18:21 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 19 Sep 2015 18:18:21 -0500
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
Message-ID: <CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>

On Sat, Sep 19, 2015 at 6:10 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> It gives zero pivots on:
>

Send -ksp_view

  Matt


> -ksp_type gmres -pc_type gamg -mg_coarse_sub_pc_factor_shift_type NONZERO
>
> It has no problem with:
>
> -ksp_type gmres -pc_type bjacobi -sub_pc_factor_shift_type NONZERO
>
> -gideon
>
> > On Sep 19, 2015, at 12:37 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >
> >> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> >>
> >> I?m getting a zero pivot in LU factorization error when I use the gamg
> preconditioner.  I know there is some issue with my matrix, as I get the
> same error if I use pc_type lu in serial, or if I use pc_type bjacobi in
> parallel, both of which are corrected by adding -pc_factor_shift_type
> NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t
> quite seem to figure out the analogous command to use for the gamg
> preconditioner.
> >
> >  You need to figure out the prefix to the option that is added by gamg
> to set it. One way is to run with the gamg options and -help and grep for
> factor_shift.
> >
> >   The option is like something like -mg_coarse_pc_factor_shift_type
> nonzero or -mg_coarse_sub_pc_factor_shift_type
> >
> >  Barry
> >
> >>
> >> -gideon
> >>
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150919/9beb049a/attachment.html>

From bsmith at mcs.anl.gov  Sat Sep 19 18:23:42 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 19 Sep 2015 18:23:42 -0500
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
Message-ID: <2E4FDB63-6B68-407A-82F5-2D0B05D37C0E@mcs.anl.gov>


Hmmm

./ex19 -da_refine 2 -pc_type gamg -help | grep mg_coarse | grep shift
  -mg_coarse_sub_pc_factor_shift_type <INBLOCKS> (choose one of) NONE NONZERO POSITIVE_DEFINITE INBLOCKS (PCFactorSetShiftType)
  -mg_coarse_sub_pc_factor_shift_amount <2.22045e-14>: Shift added to diagonal (PCFactorSetShiftAmount)

What do you get with -ksp_type gmres -pc_type gamg -help | grep factor_shift ?

What do you get with -ksp_type gmres -pc_type gamg  -mg_coarse_sub_pc_factor_shift_type gibberish  ?

You should get an error message saying it doesn't know the type. Otherwise it is not checking that option. Or perhaps you somehow have it set to use a LU factorization that doesn't support this option like SuperLU.

  Barry




> On Sep 19, 2015, at 6:10 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> It gives zero pivots on:
> 
> -ksp_type gmres -pc_type gamg -mg_coarse_sub_pc_factor_shift_type NONZERO
> 
> It has no problem with:
> 
> -ksp_type gmres -pc_type bjacobi -sub_pc_factor_shift_type NONZERO
> 
> -gideon
> 
>> On Sep 19, 2015, at 12:37 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> I?m getting a zero pivot in LU factorization error when I use the gamg preconditioner.  I know there is some issue with my matrix, as I get the same error if I use pc_type lu in serial, or if I use pc_type bjacobi in parallel, both of which are corrected by adding -pc_factor_shift_type NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t quite seem to figure out the analogous command to use for the gamg preconditioner.
>> 
>> You need to figure out the prefix to the option that is added by gamg to set it. One way is to run with the gamg options and -help and grep for factor_shift.   
>> 
>>  The option is like something like -mg_coarse_pc_factor_shift_type nonzero or -mg_coarse_sub_pc_factor_shift_type 
>> 
>> Barry
>> 
>>> 
>>> -gideon
>>> 
>> 
> 


From gideon.simpson at gmail.com  Sat Sep 19 19:12:46 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Sat, 19 Sep 2015 20:12:46 -0400
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
	<CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>
Message-ID: <BFD23752-02C2-40DF-A7EF-127F1F8D0048@gmail.com>

I never get to see the output.  When it runs, I get:

  0 SNES Function norm 8.405398322914e-02 
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot
[0]PETSC ERROR: Zero pivot on row 0
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
[0]PETSC ERROR: ./blowup_refine on a arch-darwin-c-debug named gs_air by gideon Sat Sep 19 20:11:44 2015
[0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
[0]PETSC ERROR: #1 MatSOR_SeqAIJ_Inode() line 2795 in /opt/petsc/src/mat/impls/aij/seq/inode.c
[0]PETSC ERROR: #2 MatSOR_MPIAIJ() line 1496 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #3 MatSOR() line 3697 in /opt/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #4 PCApply_SOR() line 37 in /opt/petsc/src/ksp/pc/impls/sor/sor.c
[0]PETSC ERROR: #5 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #6 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #7 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: #9 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
[0]PETSC ERROR: #11 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #13 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #14 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #15 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #16 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: #17 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: #18 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #19 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #20 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
[1]PETSC ERROR: #1 VecNorm_MPI() line 63 in /opt/petsc/src/vec/vec/impls/mpi/pvec2.c
[1]PETSC ERROR: #2 VecNorm() line 242 in /opt/petsc/src/vec/vec/interface/rvector.c
[1]PETSC ERROR: #3 VecNormalize() line 337 in /opt/petsc/src/vec/vec/interface/rvector.c
[1]PETSC ERROR: #4 KSPGMRESCycle() line 127 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
[1]PETSC ERROR: #5 KSPSolve_GMRES() line 236 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
[1]PETSC ERROR: #6 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
[1]PETSC ERROR: #8 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: #9 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
[1]PETSC ERROR: #10 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
[1]PETSC ERROR: #11 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: #12 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
[1]PETSC ERROR: #13 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
[1]PETSC ERROR: #14 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
[1]PETSC ERROR: #15 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: #16 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
[1]PETSC ERROR: #17 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c

and then it just sits there.

-gideon

> On Sep 19, 2015, at 7:18 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Sat, Sep 19, 2015 at 6:10 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> It gives zero pivots on:
> 
> Send -ksp_view
> 
>   Matt
>  
> -ksp_type gmres -pc_type gamg -mg_coarse_sub_pc_factor_shift_type NONZERO
> 
> It has no problem with:
> 
> -ksp_type gmres -pc_type bjacobi -sub_pc_factor_shift_type NONZERO
> 
> -gideon
> 
> > On Sep 19, 2015, at 12:37 AM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
> >
> >
> >> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> >>
> >> I?m getting a zero pivot in LU factorization error when I use the gamg preconditioner.  I know there is some issue with my matrix, as I get the same error if I use pc_type lu in serial, or if I use pc_type bjacobi in parallel, both of which are corrected by adding -pc_factor_shift_type NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t quite seem to figure out the analogous command to use for the gamg preconditioner.
> >
> >  You need to figure out the prefix to the option that is added by gamg to set it. One way is to run with the gamg options and -help and grep for factor_shift.
> >
> >   The option is like something like -mg_coarse_pc_factor_shift_type nonzero or -mg_coarse_sub_pc_factor_shift_type
> >
> >  Barry
> >
> >>
> >> -gideon
> >>
> >
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150919/1c41542f/attachment-0001.html>

From bsmith at mcs.anl.gov  Sat Sep 19 21:09:27 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 19 Sep 2015 21:09:27 -0500
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <BFD23752-02C2-40DF-A7EF-127F1F8D0048@gmail.com>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
	<CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>
	<BFD23752-02C2-40DF-A7EF-127F1F8D0048@gmail.com>
Message-ID: <EBFF6CCD-B770-4D6F-8562-3E4C5B743B27@mcs.anl.gov>


  Oh, it is a different kind of zero pivot value. There is a zero on the diagonal while it is trying to do SOR.  Should there be zeros on the diagonal of your matrix? 

   You can use -mg_levels_pc_type jacobi  and it will ignore those zero values.

   Barry


> On Sep 19, 2015, at 7:12 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I never get to see the output.  When it runs, I get:
> 
>   0 SNES Function norm 8.405398322914e-02 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot
> [0]PETSC ERROR: Zero pivot on row 0
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
> [0]PETSC ERROR: ./blowup_refine on a arch-darwin-c-debug named gs_air by gideon Sat Sep 19 20:11:44 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #1 MatSOR_SeqAIJ_Inode() line 2795 in /opt/petsc/src/mat/impls/aij/seq/inode.c
> [0]PETSC ERROR: #2 MatSOR_MPIAIJ() line 1496 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #3 MatSOR() line 3697 in /opt/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #4 PCApply_SOR() line 37 in /opt/petsc/src/ksp/pc/impls/sor/sor.c
> [0]PETSC ERROR: #5 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #6 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #7 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: #9 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
> [0]PETSC ERROR: #11 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #13 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #14 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #15 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #16 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
> [0]PETSC ERROR: #17 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: #18 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #19 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #20 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
> [1]PETSC ERROR: #1 VecNorm_MPI() line 63 in /opt/petsc/src/vec/vec/impls/mpi/pvec2.c
> [1]PETSC ERROR: #2 VecNorm() line 242 in /opt/petsc/src/vec/vec/interface/rvector.c
> [1]PETSC ERROR: #3 VecNormalize() line 337 in /opt/petsc/src/vec/vec/interface/rvector.c
> [1]PETSC ERROR: #4 KSPGMRESCycle() line 127 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 236 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [1]PETSC ERROR: #6 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
> [1]PETSC ERROR: #8 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #9 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
> [1]PETSC ERROR: #10 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
> [1]PETSC ERROR: #11 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: #12 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
> [1]PETSC ERROR: #13 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
> [1]PETSC ERROR: #14 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [1]PETSC ERROR: #15 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #16 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
> [1]PETSC ERROR: #17 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
> 
> and then it just sits there.
> 
> -gideon
> 
>> On Sep 19, 2015, at 7:18 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Sat, Sep 19, 2015 at 6:10 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> It gives zero pivots on:
>> 
>> Send -ksp_view
>> 
>>   Matt
>>  
>> -ksp_type gmres -pc_type gamg -mg_coarse_sub_pc_factor_shift_type NONZERO
>> 
>> It has no problem with:
>> 
>> -ksp_type gmres -pc_type bjacobi -sub_pc_factor_shift_type NONZERO
>> 
>> -gideon
>> 
>> > On Sep 19, 2015, at 12:37 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> >
>> >
>> >> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> >>
>> >> I?m getting a zero pivot in LU factorization error when I use the gamg preconditioner.  I know there is some issue with my matrix, as I get the same error if I use pc_type lu in serial, or if I use pc_type bjacobi in parallel, both of which are corrected by adding -pc_factor_shift_type NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t quite seem to figure out the analogous command to use for the gamg preconditioner.
>> >
>> >  You need to figure out the prefix to the option that is added by gamg to set it. One way is to run with the gamg options and -help and grep for factor_shift.
>> >
>> >   The option is like something like -mg_coarse_pc_factor_shift_type nonzero or -mg_coarse_sub_pc_factor_shift_type
>> >
>> >  Barry
>> >
>> >>
>> >> -gideon
>> >>
>> >
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 


From gideon.simpson at gmail.com  Sat Sep 19 21:14:22 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Sat, 19 Sep 2015 22:14:22 -0400
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <EBFF6CCD-B770-4D6F-8562-3E4C5B743B27@mcs.anl.gov>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
	<CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>
	<BFD23752-02C2-40DF-A7EF-127F1F8D0048@gmail.com>
	<EBFF6CCD-B770-4D6F-8562-3E4C5B743B27@mcs.anl.gov>
Message-ID: <7A67E976-EE55-449B-A6F8-DB59C03C3202@gmail.com>

Yes, I know exactly where the zero diagonal entries are.  Is there a way to compensate for that?

-gideon

> On Sep 19, 2015, at 10:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Oh, it is a different kind of zero pivot value. There is a zero on the diagonal while it is trying to do SOR.  Should there be zeros on the diagonal of your matrix? 
> 
>   You can use -mg_levels_pc_type jacobi  and it will ignore those zero values.
> 
>   Barry
> 
> 
>> On Sep 19, 2015, at 7:12 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> I never get to see the output.  When it runs, I get:
>> 
>>  0 SNES Function norm 8.405398322914e-02 
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot
>> [0]PETSC ERROR: Zero pivot on row 0
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
>> [0]PETSC ERROR: ./blowup_refine on a arch-darwin-c-debug named gs_air by gideon Sat Sep 19 20:11:44 2015
>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>> [0]PETSC ERROR: #1 MatSOR_SeqAIJ_Inode() line 2795 in /opt/petsc/src/mat/impls/aij/seq/inode.c
>> [0]PETSC ERROR: #2 MatSOR_MPIAIJ() line 1496 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
>> [0]PETSC ERROR: #3 MatSOR() line 3697 in /opt/petsc/src/mat/interface/matrix.c
>> [0]PETSC ERROR: #4 PCApply_SOR() line 37 in /opt/petsc/src/ksp/pc/impls/sor/sor.c
>> [0]PETSC ERROR: #5 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>> [0]PETSC ERROR: #6 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>> [0]PETSC ERROR: #7 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>> [0]PETSC ERROR: #9 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
>> [0]PETSC ERROR: #11 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>> [0]PETSC ERROR: #13 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>> [0]PETSC ERROR: #14 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>> [0]PETSC ERROR: #15 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>> [0]PETSC ERROR: #16 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>> [0]PETSC ERROR: #17 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>> [0]PETSC ERROR: #18 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #19 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
>> [0]PETSC ERROR: #20 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>> [1]PETSC ERROR: #1 VecNorm_MPI() line 63 in /opt/petsc/src/vec/vec/impls/mpi/pvec2.c
>> [1]PETSC ERROR: #2 VecNorm() line 242 in /opt/petsc/src/vec/vec/interface/rvector.c
>> [1]PETSC ERROR: #3 VecNormalize() line 337 in /opt/petsc/src/vec/vec/interface/rvector.c
>> [1]PETSC ERROR: #4 KSPGMRESCycle() line 127 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 236 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>> [1]PETSC ERROR: #6 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>> [1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
>> [1]PETSC ERROR: #8 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>> [1]PETSC ERROR: #9 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>> [1]PETSC ERROR: #10 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>> [1]PETSC ERROR: #11 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>> [1]PETSC ERROR: #12 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>> [1]PETSC ERROR: #13 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>> [1]PETSC ERROR: #14 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>> [1]PETSC ERROR: #15 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>> [1]PETSC ERROR: #16 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
>> [1]PETSC ERROR: #17 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>> 
>> and then it just sits there.
>> 
>> -gideon
>> 
>>> On Sep 19, 2015, at 7:18 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>> On Sat, Sep 19, 2015 at 6:10 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> It gives zero pivots on:
>>> 
>>> Send -ksp_view
>>> 
>>>  Matt
>>> 
>>> -ksp_type gmres -pc_type gamg -mg_coarse_sub_pc_factor_shift_type NONZERO
>>> 
>>> It has no problem with:
>>> 
>>> -ksp_type gmres -pc_type bjacobi -sub_pc_factor_shift_type NONZERO
>>> 
>>> -gideon
>>> 
>>>> On Sep 19, 2015, at 12:37 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>>> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> I?m getting a zero pivot in LU factorization error when I use the gamg preconditioner.  I know there is some issue with my matrix, as I get the same error if I use pc_type lu in serial, or if I use pc_type bjacobi in parallel, both of which are corrected by adding -pc_factor_shift_type NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t quite seem to figure out the analogous command to use for the gamg preconditioner.
>>>> 
>>>> You need to figure out the prefix to the option that is added by gamg to set it. One way is to run with the gamg options and -help and grep for factor_shift.
>>>> 
>>>>  The option is like something like -mg_coarse_pc_factor_shift_type nonzero or -mg_coarse_sub_pc_factor_shift_type
>>>> 
>>>> Barry
>>>> 
>>>>> 
>>>>> -gideon
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150919/72b4a5ab/attachment.html>

From bsmith at mcs.anl.gov  Sat Sep 19 21:43:42 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 19 Sep 2015 21:43:42 -0500
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <7A67E976-EE55-449B-A6F8-DB59C03C3202@gmail.com>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
	<CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>
	<BFD23752-02C2-40DF-A7EF-127F1F8D0048@gmail.com>
	<EBFF6CCD-B770-4D6F-8562-3E4C5B743B27@mcs.anl.gov>
	<7A67E976-EE55-449B-A6F8-DB59C03C3202@gmail.com>
Message-ID: <2C9E2DA5-1A12-4AFC-B24C-F166C1D1A2A6@mcs.anl.gov>


> On Sep 19, 2015, at 9:14 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> Yes, I know exactly where the zero diagonal entries are.  Is there a way to compensate for that?

  Not really. How can you update x_i if A_ii = 0 ?


> 
> -gideon
> 
>> On Sep 19, 2015, at 10:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  Oh, it is a different kind of zero pivot value. There is a zero on the diagonal while it is trying to do SOR.  Should there be zeros on the diagonal of your matrix? 
>> 
>>   You can use -mg_levels_pc_type jacobi  and it will ignore those zero values.
>> 
>>   Barry
>> 
>> 
>>> On Sep 19, 2015, at 7:12 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> I never get to see the output.  When it runs, I get:
>>> 
>>>  0 SNES Function norm 8.405398322914e-02 
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot
>>> [0]PETSC ERROR: Zero pivot on row 0
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
>>> [0]PETSC ERROR: ./blowup_refine on a arch-darwin-c-debug named gs_air by gideon Sat Sep 19 20:11:44 2015
>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>> [0]PETSC ERROR: #1 MatSOR_SeqAIJ_Inode() line 2795 in /opt/petsc/src/mat/impls/aij/seq/inode.c
>>> [0]PETSC ERROR: #2 MatSOR_MPIAIJ() line 1496 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
>>> [0]PETSC ERROR: #3 MatSOR() line 3697 in /opt/petsc/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #4 PCApply_SOR() line 37 in /opt/petsc/src/ksp/pc/impls/sor/sor.c
>>> [0]PETSC ERROR: #5 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>>> [0]PETSC ERROR: #6 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>>> [0]PETSC ERROR: #7 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>>> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: #9 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
>>> [0]PETSC ERROR: #11 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>> [0]PETSC ERROR: #13 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>> [0]PETSC ERROR: #14 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>>> [0]PETSC ERROR: #15 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>>> [0]PETSC ERROR: #16 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>>> [0]PETSC ERROR: #17 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: #18 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: #19 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
>>> [0]PETSC ERROR: #20 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>>> [1]PETSC ERROR: #1 VecNorm_MPI() line 63 in /opt/petsc/src/vec/vec/impls/mpi/pvec2.c
>>> [1]PETSC ERROR: #2 VecNorm() line 242 in /opt/petsc/src/vec/vec/interface/rvector.c
>>> [1]PETSC ERROR: #3 VecNormalize() line 337 in /opt/petsc/src/vec/vec/interface/rvector.c
>>> [1]PETSC ERROR: #4 KSPGMRESCycle() line 127 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 236 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>> [1]PETSC ERROR: #6 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>> [1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
>>> [1]PETSC ERROR: #8 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>> [1]PETSC ERROR: #9 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>> [1]PETSC ERROR: #10 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>> [1]PETSC ERROR: #11 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>>> [1]PETSC ERROR: #12 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>>> [1]PETSC ERROR: #13 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>>> [1]PETSC ERROR: #14 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>> [1]PETSC ERROR: #15 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>> [1]PETSC ERROR: #16 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
>>> [1]PETSC ERROR: #17 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>>> 
>>> and then it just sits there.
>>> 
>>> -gideon
>>> 
>>>> On Sep 19, 2015, at 7:18 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>> 
>>>> On Sat, Sep 19, 2015 at 6:10 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> It gives zero pivots on:
>>>> 
>>>> Send -ksp_view
>>>> 
>>>>  Matt
>>>> 
>>>> -ksp_type gmres -pc_type gamg -mg_coarse_sub_pc_factor_shift_type NONZERO
>>>> 
>>>> It has no problem with:
>>>> 
>>>> -ksp_type gmres -pc_type bjacobi -sub_pc_factor_shift_type NONZERO
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 19, 2015, at 12:37 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> 
>>>>>> I?m getting a zero pivot in LU factorization error when I use the gamg preconditioner.  I know there is some issue with my matrix, as I get the same error if I use pc_type lu in serial, or if I use pc_type bjacobi in parallel, both of which are corrected by adding -pc_factor_shift_type NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t quite seem to figure out the analogous command to use for the gamg preconditioner.
>>>>> 
>>>>> You need to figure out the prefix to the option that is added by gamg to set it. One way is to run with the gamg options and -help and grep for factor_shift.
>>>>> 
>>>>>  The option is like something like -mg_coarse_pc_factor_shift_type nonzero or -mg_coarse_sub_pc_factor_shift_type
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>> 
> 


From gideon.simpson at gmail.com  Sun Sep 20 12:40:23 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Sun, 20 Sep 2015 13:40:23 -0400
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <2C9E2DA5-1A12-4AFC-B24C-F166C1D1A2A6@mcs.anl.gov>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
	<CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>
	<BFD23752-02C2-40DF-A7EF-127F1F8D0048@gmail.com>
	<EBFF6CCD-B770-4D6F-8562-3E4C5B743B27@mcs.anl.gov>
	<7A67E976-EE55-449B-A6F8-DB59C03C3202@gmail.com>
	<2C9E2DA5-1A12-4AFC-B24C-F166C1D1A2A6@mcs.anl.gov>
Message-ID: <691C9467-BC7D-4911-AF48-78085128584D@gmail.com>

Point taken.  So this comes back to something that I began asking about in a previous thread, but took a detour as you guys identified other problems with my code.  The issue is that I have a DMComposite with two blocks.  One block is of size two, and couples certain pieces of the second block, which is, morally, a vector valued semilinear elliptic equation.  Using grid sequencing, the guesses for the SNES are now quite good, but I?m struggling with the issue of preconditioning the linear problem.  The second block would, presumably, benefit from multigrid, but jacobi doesn?t do such a good job on that. At the same time, the first block, as discussed, has these zeros on the diagonal, and can?t use SOR.

-gideon

> On Sep 19, 2015, at 10:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>> On Sep 19, 2015, at 9:14 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>> 
>> Yes, I know exactly where the zero diagonal entries are.  Is there a way to compensate for that?
> 
>  Not really. How can you update x_i if A_ii = 0 ?
> 
> 
>> 
>> -gideon
>> 
>>> On Sep 19, 2015, at 10:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> Oh, it is a different kind of zero pivot value. There is a zero on the diagonal while it is trying to do SOR.  Should there be zeros on the diagonal of your matrix? 
>>> 
>>>  You can use -mg_levels_pc_type jacobi  and it will ignore those zero values.
>>> 
>>>  Barry
>>> 
>>> 
>>>> On Sep 19, 2015, at 7:12 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>> 
>>>> I never get to see the output.  When it runs, I get:
>>>> 
>>>> 0 SNES Function norm 8.405398322914e-02 
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot
>>>> [0]PETSC ERROR: Zero pivot on row 0
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
>>>> [0]PETSC ERROR: ./blowup_refine on a arch-darwin-c-debug named gs_air by gideon Sat Sep 19 20:11:44 2015
>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>> [0]PETSC ERROR: #1 MatSOR_SeqAIJ_Inode() line 2795 in /opt/petsc/src/mat/impls/aij/seq/inode.c
>>>> [0]PETSC ERROR: #2 MatSOR_MPIAIJ() line 1496 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
>>>> [0]PETSC ERROR: #3 MatSOR() line 3697 in /opt/petsc/src/mat/interface/matrix.c
>>>> [0]PETSC ERROR: #4 PCApply_SOR() line 37 in /opt/petsc/src/ksp/pc/impls/sor/sor.c
>>>> [0]PETSC ERROR: #5 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>>>> [0]PETSC ERROR: #6 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>>>> [0]PETSC ERROR: #7 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>>>> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>>> [0]PETSC ERROR: #9 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>> [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
>>>> [0]PETSC ERROR: #11 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>> [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>>> [0]PETSC ERROR: #13 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>>> [0]PETSC ERROR: #14 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>>>> [0]PETSC ERROR: #15 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>>>> [0]PETSC ERROR: #16 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>>>> [0]PETSC ERROR: #17 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>>> [0]PETSC ERROR: #18 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>> [0]PETSC ERROR: #19 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
>>>> [0]PETSC ERROR: #20 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>>>> [1]PETSC ERROR: #1 VecNorm_MPI() line 63 in /opt/petsc/src/vec/vec/impls/mpi/pvec2.c
>>>> [1]PETSC ERROR: #2 VecNorm() line 242 in /opt/petsc/src/vec/vec/interface/rvector.c
>>>> [1]PETSC ERROR: #3 VecNormalize() line 337 in /opt/petsc/src/vec/vec/interface/rvector.c
>>>> [1]PETSC ERROR: #4 KSPGMRESCycle() line 127 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 236 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>>> [1]PETSC ERROR: #6 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>> [1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
>>>> [1]PETSC ERROR: #8 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>> [1]PETSC ERROR: #9 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>>> [1]PETSC ERROR: #10 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>>> [1]PETSC ERROR: #11 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>>>> [1]PETSC ERROR: #12 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>>>> [1]PETSC ERROR: #13 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>>>> [1]PETSC ERROR: #14 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>>> [1]PETSC ERROR: #15 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>> [1]PETSC ERROR: #16 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
>>>> [1]PETSC ERROR: #17 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>>>> 
>>>> and then it just sits there.
>>>> 
>>>> -gideon
>>>> 
>>>>> On Sep 19, 2015, at 7:18 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>> 
>>>>> On Sat, Sep 19, 2015 at 6:10 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> It gives zero pivots on:
>>>>> 
>>>>> Send -ksp_view
>>>>> 
>>>>> Matt
>>>>> 
>>>>> -ksp_type gmres -pc_type gamg -mg_coarse_sub_pc_factor_shift_type NONZERO
>>>>> 
>>>>> It has no problem with:
>>>>> 
>>>>> -ksp_type gmres -pc_type bjacobi -sub_pc_factor_shift_type NONZERO
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 19, 2015, at 12:37 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>>> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>> 
>>>>>>> I?m getting a zero pivot in LU factorization error when I use the gamg preconditioner.  I know there is some issue with my matrix, as I get the same error if I use pc_type lu in serial, or if I use pc_type bjacobi in parallel, both of which are corrected by adding -pc_factor_shift_type NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t quite seem to figure out the analogous command to use for the gamg preconditioner.
>>>>>> 
>>>>>> You need to figure out the prefix to the option that is added by gamg to set it. One way is to run with the gamg options and -help and grep for factor_shift.
>>>>>> 
>>>>>> The option is like something like -mg_coarse_pc_factor_shift_type nonzero or -mg_coarse_sub_pc_factor_shift_type
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>>> 
>>>>>>> -gideon
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150920/48ec56aa/attachment.html>

From bsmith at mcs.anl.gov  Sun Sep 20 12:57:05 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 20 Sep 2015 12:57:05 -0500
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <691C9467-BC7D-4911-AF48-78085128584D@gmail.com>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
	<CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>
	<BFD23752-02C2-40DF-A7EF-127F1F8D0048@gmail.com>
	<EBFF6CCD-B770-4D6F-8562-3E4C5B743B27@mcs.anl.gov>
	<7A67E976-EE55-449B-A6F8-DB59C03C3202@gmail.com>
	<2C9E2DA5-1A12-4AFC-B24C-F166C1D1A2A6@mcs.anl.gov>
	<691C9467-BC7D-4911-AF48-78085128584D@gmail.com>
Message-ID: <7FDB9D2A-A598-4F55-8395-D4214A1B0B6C@mcs.anl.gov>


> On Sep 20, 2015, at 12:40 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> Point taken.  So this comes back to something that I began asking about in a previous thread, but took a detour as you guys identified other problems with my code.  The issue is that I have a DMComposite with two blocks.  One block is of size two, and couples certain pieces of the second block, which is, morally, a vector valued semilinear elliptic equation.  Using grid sequencing, the guesses for the SNES are now quite good, but I?m struggling with the issue of preconditioning the linear problem.  The second block would, presumably, benefit from multigrid, but jacobi doesn?t do such a good job on that. At the same time, the first block, as discussed, has these zeros on the diagonal, and can?t use SOR.

   I think we answered this before, this is what PCFIELDSPLIT is for.  If the matrix looks like (logically if you reorder rows and columns)

(A   B )
(C  0  )

  you should use something like -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_detect_saddle_point -pc_fieldsplit_type schur  -fieldsplit_0_ksp_type gmres  -fieldsplit_0_pc_type gamg 

the detect saddle point flag automatically makes the first block fieldsplit_0 the block without the zeros on the diagonal and the second block fieldsplit_1 be the one with zeros on the diagonal.  You will likely need to play around a bit with sub options to get great convergence but 
start by running with -ksp_monitor -fieldsplit_0_ksp_monitor_true_residual and see how it converges.

  Barry

> 
> -gideon
> 
>> On Sep 19, 2015, at 10:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>> On Sep 19, 2015, at 9:14 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>> 
>>> Yes, I know exactly where the zero diagonal entries are.  Is there a way to compensate for that?
>> 
>>  Not really. How can you update x_i if A_ii = 0 ?
>> 
>> 
>>> 
>>> -gideon
>>> 
>>>> On Sep 19, 2015, at 10:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> Oh, it is a different kind of zero pivot value. There is a zero on the diagonal while it is trying to do SOR.  Should there be zeros on the diagonal of your matrix? 
>>>> 
>>>>  You can use -mg_levels_pc_type jacobi  and it will ignore those zero values.
>>>> 
>>>>  Barry
>>>> 
>>>> 
>>>>> On Sep 19, 2015, at 7:12 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>> 
>>>>> I never get to see the output.  When it runs, I get:
>>>>> 
>>>>> 0 SNES Function norm 8.405398322914e-02 
>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>> [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot
>>>>> [0]PETSC ERROR: Zero pivot on row 0
>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown 
>>>>> [0]PETSC ERROR: ./blowup_refine on a arch-darwin-c-debug named gs_air by gideon Sat Sep 19 20:11:44 2015
>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>> [0]PETSC ERROR: #1 MatSOR_SeqAIJ_Inode() line 2795 in /opt/petsc/src/mat/impls/aij/seq/inode.c
>>>>> [0]PETSC ERROR: #2 MatSOR_MPIAIJ() line 1496 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
>>>>> [0]PETSC ERROR: #3 MatSOR() line 3697 in /opt/petsc/src/mat/interface/matrix.c
>>>>> [0]PETSC ERROR: #4 PCApply_SOR() line 37 in /opt/petsc/src/ksp/pc/impls/sor/sor.c
>>>>> [0]PETSC ERROR: #5 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>>>>> [0]PETSC ERROR: #6 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>>>>> [0]PETSC ERROR: #7 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>>>>> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>>>> [0]PETSC ERROR: #9 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
>>>>> [0]PETSC ERROR: #11 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>>>> [0]PETSC ERROR: #13 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>>>> [0]PETSC ERROR: #14 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>>>>> [0]PETSC ERROR: #15 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>>>>> [0]PETSC ERROR: #16 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>>>>> [0]PETSC ERROR: #17 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>>>> [0]PETSC ERROR: #18 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> [0]PETSC ERROR: #19 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
>>>>> [0]PETSC ERROR: #20 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>>>>> [1]PETSC ERROR: #1 VecNorm_MPI() line 63 in /opt/petsc/src/vec/vec/impls/mpi/pvec2.c
>>>>> [1]PETSC ERROR: #2 VecNorm() line 242 in /opt/petsc/src/vec/vec/interface/rvector.c
>>>>> [1]PETSC ERROR: #3 VecNormalize() line 337 in /opt/petsc/src/vec/vec/interface/rvector.c
>>>>> [1]PETSC ERROR: #4 KSPGMRESCycle() line 127 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 236 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>>>> [1]PETSC ERROR: #6 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> [1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 381 in /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
>>>>> [1]PETSC ERROR: #8 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> [1]PETSC ERROR: #9 PCMGMCycle_Private() line 19 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>>>> [1]PETSC ERROR: #10 PCApply_MG() line 340 in /opt/petsc/src/ksp/pc/impls/mg/mg.c
>>>>> [1]PETSC ERROR: #11 PCApply() line 483 in /opt/petsc/src/ksp/pc/interface/precon.c
>>>>> [1]PETSC ERROR: #12 KSP_PCApply() line 242 in /opt/petsc/include/petsc/private/kspimpl.h
>>>>> [1]PETSC ERROR: #13 KSPInitialResidual() line 63 in /opt/petsc/src/ksp/ksp/interface/itres.c
>>>>> [1]PETSC ERROR: #14 KSPSolve_GMRES() line 235 in /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>>>> [1]PETSC ERROR: #15 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> [1]PETSC ERROR: #16 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
>>>>> [1]PETSC ERROR: #17 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
>>>>> 
>>>>> and then it just sits there.
>>>>> 
>>>>> -gideon
>>>>> 
>>>>>> On Sep 19, 2015, at 7:18 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>> 
>>>>>> On Sat, Sep 19, 2015 at 6:10 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> It gives zero pivots on:
>>>>>> 
>>>>>> Send -ksp_view
>>>>>> 
>>>>>> Matt
>>>>>> 
>>>>>> -ksp_type gmres -pc_type gamg -mg_coarse_sub_pc_factor_shift_type NONZERO
>>>>>> 
>>>>>> It has no problem with:
>>>>>> 
>>>>>> -ksp_type gmres -pc_type bjacobi -sub_pc_factor_shift_type NONZERO
>>>>>> 
>>>>>> -gideon
>>>>>> 
>>>>>>> On Sep 19, 2015, at 12:37 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> I?m getting a zero pivot in LU factorization error when I use the gamg preconditioner.  I know there is some issue with my matrix, as I get the same error if I use pc_type lu in serial, or if I use pc_type bjacobi in parallel, both of which are corrected by adding -pc_factor_shift_type NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t quite seem to figure out the analogous command to use for the gamg preconditioner.
>>>>>>> 
>>>>>>> You need to figure out the prefix to the option that is added by gamg to set it. One way is to run with the gamg options and -help and grep for factor_shift.
>>>>>>> 
>>>>>>> The option is like something like -mg_coarse_pc_factor_shift_type nonzero or -mg_coarse_sub_pc_factor_shift_type
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>>> 
>>>>>>>> -gideon
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>> 
>>>> 
>>> 
>> 
> 


From knepley at gmail.com  Sun Sep 20 13:57:51 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 20 Sep 2015 13:57:51 -0500
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <691C9467-BC7D-4911-AF48-78085128584D@gmail.com>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
	<CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>
	<BFD23752-02C2-40DF-A7EF-127F1F8D0048@gmail.com>
	<EBFF6CCD-B770-4D6F-8562-3E4C5B743B27@mcs.anl.gov>
	<7A67E976-EE55-449B-A6F8-DB59C03C3202@gmail.com>
	<2C9E2DA5-1A12-4AFC-B24C-F166C1D1A2A6@mcs.anl.gov>
	<691C9467-BC7D-4911-AF48-78085128584D@gmail.com>
Message-ID: <CAMYG4Gne4oMRrxKxQv=n8h0bMWy2thjACAgvyd9xmxvE-7RgBA@mail.gmail.com>

On Sun, Sep 20, 2015 at 12:40 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Point taken.  So this comes back to something that I began asking about in
> a previous thread, but took a detour as you guys identified other problems
> with my code.  The issue is that I have a DMComposite with two blocks.  One
> block is of size two, and couples certain pieces of the second block, which
> is, morally, a vector valued semilinear elliptic equation.  Using grid
> sequencing, the guesses for the SNES are now quite good, but I?m struggling
> with the issue of preconditioning the linear problem.  The second block
> would, presumably, benefit from multigrid, but jacobi doesn?t do such a
> good job on that. At the same time, the first block, as discussed, has
> these zeros on the diagonal, and can?t use SOR.
>

Chebyshev/Jacobi is the obvious next try.

   Matt


> -gideon
>
> On Sep 19, 2015, at 10:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Sep 19, 2015, at 9:14 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> Yes, I know exactly where the zero diagonal entries are.  Is there a way
> to compensate for that?
>
>
>  Not really. How can you update x_i if A_ii = 0 ?
>
>
>
> -gideon
>
> On Sep 19, 2015, at 10:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Oh, it is a different kind of zero pivot value. There is a zero on the
> diagonal while it is trying to do SOR.  Should there be zeros on the
> diagonal of your matrix?
>
>  You can use -mg_levels_pc_type jacobi  and it will ignore those zero
> values.
>
>  Barry
>
>
> On Sep 19, 2015, at 7:12 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> I never get to see the output.  When it runs, I get:
>
> 0 SNES Function norm 8.405398322914e-02
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Zero pivot in LU factorization:
> http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot
> [0]PETSC ERROR: Zero pivot on row 0
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown
> [0]PETSC ERROR: ./blowup_refine on a arch-darwin-c-debug named gs_air by
> gideon Sat Sep 19 20:11:44 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes
> --download-suitesparse=yes --download-superlu=yes
> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes
> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes
> --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #1 MatSOR_SeqAIJ_Inode() line 2795 in
> /opt/petsc/src/mat/impls/aij/seq/inode.c
> [0]PETSC ERROR: #2 MatSOR_MPIAIJ() line 1496 in
> /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #3 MatSOR() line 3697 in
> /opt/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #4 PCApply_SOR() line 37 in
> /opt/petsc/src/ksp/pc/impls/sor/sor.c
> [0]PETSC ERROR: #5 PCApply() line 483 in
> /opt/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #6 KSP_PCApply() line 242 in
> /opt/petsc/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #7 KSPInitialResidual() line 63 in
> /opt/petsc/src/ksp/ksp/interface/itres.c
> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in
> /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: #9 KSPSolve() line 604 in
> /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in
> /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
> [0]PETSC ERROR: #11 KSPSolve() line 604 in
> /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in
> /opt/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #13 PCApply_MG() line 340 in
> /opt/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #14 PCApply() line 483 in
> /opt/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #15 KSP_PCApply() line 242 in
> /opt/petsc/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #16 KSPInitialResidual() line 63 in
> /opt/petsc/src/ksp/ksp/interface/itres.c
> [0]PETSC ERROR: #17 KSPSolve_GMRES() line 235 in
> /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: #18 KSPSolve() line 604 in
> /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #19 SNESSolve_NEWTONLS() line 233 in
> /opt/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #20 SNESSolve() line 3906 in
> /opt/petsc/src/snes/interface/snes.c
> [1]PETSC ERROR: #1 VecNorm_MPI() line 63 in
> /opt/petsc/src/vec/vec/impls/mpi/pvec2.c
> [1]PETSC ERROR: #2 VecNorm() line 242 in
> /opt/petsc/src/vec/vec/interface/rvector.c
> [1]PETSC ERROR: #3 VecNormalize() line 337 in
> /opt/petsc/src/vec/vec/interface/rvector.c
> [1]PETSC ERROR: #4 KSPGMRESCycle() line 127 in
> /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 236 in
> /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [1]PETSC ERROR: #6 KSPSolve() line 604 in
> /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 381 in
> /opt/petsc/src/ksp/ksp/impls/cheby/cheby.c
> [1]PETSC ERROR: #8 KSPSolve() line 604 in
> /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #9 PCMGMCycle_Private() line 19 in
> /opt/petsc/src/ksp/pc/impls/mg/mg.c
> [1]PETSC ERROR: #10 PCApply_MG() line 340 in
> /opt/petsc/src/ksp/pc/impls/mg/mg.c
> [1]PETSC ERROR: #11 PCApply() line 483 in
> /opt/petsc/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: #12 KSP_PCApply() line 242 in
> /opt/petsc/include/petsc/private/kspimpl.h
> [1]PETSC ERROR: #13 KSPInitialResidual() line 63 in
> /opt/petsc/src/ksp/ksp/interface/itres.c
> [1]PETSC ERROR: #14 KSPSolve_GMRES() line 235 in
> /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [1]PETSC ERROR: #15 KSPSolve() line 604 in
> /opt/petsc/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #16 SNESSolve_NEWTONLS() line 233 in
> /opt/petsc/src/snes/impls/ls/ls.c
> [1]PETSC ERROR: #17 SNESSolve() line 3906 in
> /opt/petsc/src/snes/interface/snes.c
>
> and then it just sits there.
>
> -gideon
>
> On Sep 19, 2015, at 7:18 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Sat, Sep 19, 2015 at 6:10 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> It gives zero pivots on:
>
> Send -ksp_view
>
> Matt
>
> -ksp_type gmres -pc_type gamg -mg_coarse_sub_pc_factor_shift_type NONZERO
>
> It has no problem with:
>
> -ksp_type gmres -pc_type bjacobi -sub_pc_factor_shift_type NONZERO
>
> -gideon
>
> On Sep 19, 2015, at 12:37 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Sep 18, 2015, at 10:42 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
>
> I?m getting a zero pivot in LU factorization error when I use the gamg
> preconditioner.  I know there is some issue with my matrix, as I get the
> same error if I use pc_type lu in serial, or if I use pc_type bjacobi in
> parallel, both of which are corrected by adding -pc_factor_shift_type
> NONZERO -sub_pc_factor_shift_type NONZERO, respectively.  However, I can?t
> quite seem to figure out the analogous command to use for the gamg
> preconditioner.
>
>
> You need to figure out the prefix to the option that is added by gamg to
> set it. One way is to run with the gamg options and -help and grep for
> factor_shift.
>
> The option is like something like -mg_coarse_pc_factor_shift_type nonzero
> or -mg_coarse_sub_pc_factor_shift_type
>
> Barry
>
>
> -gideon
>
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150920/4aebdb4e/attachment-0001.html>

From mfadams at lbl.gov  Mon Sep 21 07:21:03 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 21 Sep 2015 08:21:03 -0400
Subject: [petsc-users] Hypre's BoomerAMG vs PETSc's GAMG?
In-Reply-To: <87zj0k6jti.fsf@jedbrown.org>
References: <CAP2=TMhGpE8UkvpTuB7uuvcwRU=4Jhb+at=oyiaRgPyz7pkZ5A@mail.gmail.com>
	<87zj0k6jti.fsf@jedbrown.org>
Message-ID: <CADOhEh6D5wcLCmiHqDL2S1qZ+7VgvA_bQdMgQxzGJk8-vXCLXQ@mail.gmail.com>

And I will just add that performance is sensitive to parameters. The
defaults try to be conservative and hypre's seem to be geared for 2D low
order discretizations.  GAMG is probably a bit more geared for 3D.  If you
are interested in looking this carefully you can run GAMG with '-info', and
grep on GAMG, and send us the results and we can verify that it running OK.
Also run with -log_summary and send that separately.  The two solvers
should be about the same speed on your problem.


On Fri, Sep 18, 2015 at 7:36 AM, Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang <jychang48 at gmail.com> writes:
> > I see that there's GAMG and there's Hypre's BoomerAMG (and perhaps others
> > too?)
>
> ML
>
> > What exactly is the difference between these two? Do they have very
> > different implementations under the hood? Does one have better
> > scalability over another? Or more importantly, when would I choose one
> > over the other?
>
> BoomerAMG is classical AMG, which is a different coarsening strategy
> From GAMG and ML which use smoothed aggregation.  The math is different
> and the implementation is different.  ML and GAMG have similar math and
> there exist configurations that are equivalent.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150921/ab40496f/attachment.html>

From jed at jedbrown.org  Mon Sep 21 08:21:35 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 21 Sep 2015 07:21:35 -0600
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <CAMYG4Gne4oMRrxKxQv=n8h0bMWy2thjACAgvyd9xmxvE-7RgBA@mail.gmail.com>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
	<CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>
	<BFD23752-02C2-40DF-A7EF-127F1F8D0048@gmail.com>
	<EBFF6CCD-B770-4D6F-8562-3E4C5B743B27@mcs.anl.gov>
	<7A67E976-EE55-449B-A6F8-DB59C03C3202@gmail.com>
	<2C9E2DA5-1A12-4AFC-B24C-F166C1D1A2A6@mcs.anl.gov>
	<691C9467-BC7D-4911-AF48-78085128584D@gmail.com>
	<CAMYG4Gne4oMRrxKxQv=n8h0bMWy2thjACAgvyd9xmxvE-7RgBA@mail.gmail.com>
Message-ID: <87bncv6h7k.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
>>At the same time, the first block, as discussed, has
>> these zeros on the diagonal, and can?t use SOR.
>>
>
> Chebyshev/Jacobi is the obvious next try.

Not really; the Chebyshev polynomial exceeds 1 left of the origin, so
it's pretty non-functional on indefinite operators.  Pretending like
Gideon's saddle point problem is positive definite is not a solution.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150921/723391c8/attachment.pgp>

From gideon.simpson at gmail.com  Mon Sep 21 08:44:43 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Mon, 21 Sep 2015 09:44:43 -0400
Subject: [petsc-users] gamg and zero pivots
In-Reply-To: <87bncv6h7k.fsf@jedbrown.org>
References: <26F03DBA-B3CB-407D-A61C-66F75283A1DB@gmail.com>
	<9EE73F4F-8A78-400C-A357-4F13224DA5EC@mcs.anl.gov>
	<88A372C5-E67D-4874-910C-C7786EA43B45@gmail.com>
	<CAMYG4G=OrsSOKmu0Yk=fiaE-EYCKYhkQ=hkBUhjRf7TQHXoFRA@mail.gmail.com>
	<BFD23752-02C2-40DF-A7EF-127F1F8D0048@gmail.com>
	<EBFF6CCD-B770-4D6F-8562-3E4C5B743B27@mcs.anl.gov>
	<7A67E976-EE55-449B-A6F8-DB59C03C3202@gmail.com>
	<2C9E2DA5-1A12-4AFC-B24C-F166C1D1A2A6@mcs.anl.gov>
	<691C9467-BC7D-4911-AF48-78085128584D@gmail.com>
	<CAMYG4Gne4oMRrxKxQv=n8h0bMWy2thjACAgvyd9xmxvE-7RgBA@mail.gmail.com>
	<87bncv6h7k.fsf@jedbrown.org>
Message-ID: <194022FE-0D60-4851-94D8-C3622A40EC4E@gmail.com>

Thanks for the suggestions guys.  Barry?s field split approach gives sensible behavior.  Now I?m working on tuning it.
-gideon

> On Sep 21, 2015, at 9:21 AM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Matthew Knepley <knepley at gmail.com> writes:
>>> At the same time, the first block, as discussed, has
>>> these zeros on the diagonal, and can?t use SOR.
>>> 
>> 
>> Chebyshev/Jacobi is the obvious next try.
> 
> Not really; the Chebyshev polynomial exceeds 1 left of the origin, so
> it's pretty non-functional on indefinite operators.  Pretending like
> Gideon's saddle point problem is positive definite is not a solution.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150921/0ac81913/attachment.html>

From jychang48 at gmail.com  Mon Sep 21 14:25:32 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Mon, 21 Sep 2015 13:25:32 -0600
Subject: [petsc-users] Hypre's BoomerAMG vs PETSc's GAMG?
In-Reply-To: <CADOhEh6D5wcLCmiHqDL2S1qZ+7VgvA_bQdMgQxzGJk8-vXCLXQ@mail.gmail.com>
References: <CAP2=TMhGpE8UkvpTuB7uuvcwRU=4Jhb+at=oyiaRgPyz7pkZ5A@mail.gmail.com>
	<87zj0k6jti.fsf@jedbrown.org>
	<CADOhEh6D5wcLCmiHqDL2S1qZ+7VgvA_bQdMgQxzGJk8-vXCLXQ@mail.gmail.com>
Message-ID: <CAP2=TMhsZ+hn-_8r0YBH5Y5R+Ow+C+tS5SN-PC--Hbrc+aSbmw@mail.gmail.com>

Okay I will just experiment around with these then. Thanks for the input
everyone

On Mon, Sep 21, 2015 at 6:21 AM, Mark Adams <mfadams at lbl.gov> wrote:

> And I will just add that performance is sensitive to parameters. The
> defaults try to be conservative and hypre's seem to be geared for 2D low
> order discretizations.  GAMG is probably a bit more geared for 3D.  If you
> are interested in looking this carefully you can run GAMG with '-info', and
> grep on GAMG, and send us the results and we can verify that it running OK.
> Also run with -log_summary and send that separately.  The two solvers
> should be about the same speed on your problem.
>
>
> On Fri, Sep 18, 2015 at 7:36 AM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Justin Chang <jychang48 at gmail.com> writes:
>> > I see that there's GAMG and there's Hypre's BoomerAMG (and perhaps
>> others
>> > too?)
>>
>> ML
>>
>> > What exactly is the difference between these two? Do they have very
>> > different implementations under the hood? Does one have better
>> > scalability over another? Or more importantly, when would I choose one
>> > over the other?
>>
>> BoomerAMG is classical AMG, which is a different coarsening strategy
>> From GAMG and ML which use smoothed aggregation.  The math is different
>> and the implementation is different.  ML and GAMG have similar math and
>> there exist configurations that are equivalent.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150921/440c2225/attachment.html>

From jychang48 at gmail.com  Mon Sep 21 14:35:12 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Mon, 21 Sep 2015 13:35:12 -0600
Subject: [petsc-users] VecChop not in petsc4py
Message-ID: <CAP2=TMiD+AMQf-MuE9kHtEg1rhj29wN0wrVq4VvV2zVeoTV4JA@mail.gmail.com>

Hi all,

I noticed that there's no VecChop in petsc4py. I see a MatChop, but was
wondering if you guys can add the petsc4py implementation of VecChop.

I am guessing it would be something like:

in src/PETSc/Vec.pyx:

def chop(self, tol):
  cdef PetscReal rval = asReal(tol)
  CHKERR( VecChop(self.vec, rval) )

in src/PETSc/petscvec.pxi:

int VecChop(PetscVec, PetscReal)

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150921/449c31f5/attachment.html>

From jed at jedbrown.org  Mon Sep 21 14:53:08 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 21 Sep 2015 13:53:08 -0600
Subject: [petsc-users] VecChop not in petsc4py
In-Reply-To: <CAP2=TMiD+AMQf-MuE9kHtEg1rhj29wN0wrVq4VvV2zVeoTV4JA@mail.gmail.com>
References: <CAP2=TMiD+AMQf-MuE9kHtEg1rhj29wN0wrVq4VvV2zVeoTV4JA@mail.gmail.com>
Message-ID: <877fnj4kij.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:

> Hi all,
>
> I noticed that there's no VecChop in petsc4py. I see a MatChop, but was
> wondering if you guys can add the petsc4py implementation of VecChop.
>
> I am guessing it would be something like:

Since you've done the work, you might as well submit it as a pull
request.

> in src/PETSc/Vec.pyx:
>
> def chop(self, tol):
>   cdef PetscReal rval = asReal(tol)
>   CHKERR( VecChop(self.vec, rval) )
>
> in src/PETSc/petscvec.pxi:
>
> int VecChop(PetscVec, PetscReal)
>
> Thanks,
> Justin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150921/db696661/attachment.pgp>

From jychang48 at gmail.com  Mon Sep 21 18:15:04 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Mon, 21 Sep 2015 17:15:04 -0600
Subject: [petsc-users] VecChop not in petsc4py
In-Reply-To: <877fnj4kij.fsf@jedbrown.org>
References: <CAP2=TMiD+AMQf-MuE9kHtEg1rhj29wN0wrVq4VvV2zVeoTV4JA@mail.gmail.com>
	<877fnj4kij.fsf@jedbrown.org>
Message-ID: <CAP2=TMj0VOokBae5CgO8fXhNYWfwFwxDQy2F+DkFM8uZvTUGOQ@mail.gmail.com>

Just did, it works for my test problems, so hope it gets passed ;)

On Mon, Sep 21, 2015 at 1:53 PM, Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang <jychang48 at gmail.com> writes:
>
> > Hi all,
> >
> > I noticed that there's no VecChop in petsc4py. I see a MatChop, but was
> > wondering if you guys can add the petsc4py implementation of VecChop.
> >
> > I am guessing it would be something like:
>
> Since you've done the work, you might as well submit it as a pull
> request.
>
> > in src/PETSc/Vec.pyx:
> >
> > def chop(self, tol):
> >   cdef PetscReal rval = asReal(tol)
> >   CHKERR( VecChop(self.vec, rval) )
> >
> > in src/PETSc/petscvec.pxi:
> >
> > int VecChop(PetscVec, PetscReal)
> >
> > Thanks,
> > Justin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150921/faf07efc/attachment.html>

From soumyamechanics at gmail.com  Mon Sep 21 22:59:20 2015
From: soumyamechanics at gmail.com (Soumya Mukherjee)
Date: Mon, 21 Sep 2015 23:59:20 -0400
Subject: [petsc-users] PETSC error: Caught signal number 8 FPE
Message-ID: <CAPuCtQFUhurjiQisYOketmvwmG0UONNBPA1711oEBfLWisp0gQ@mail.gmail.com>

Hi.

I am trying to solve an eigenvalue problem. I have a code that works for a
particular input file. And, whenever I change the input file, the following
error message appears. Any hint for debugging, please?

PETSC ERROR:
------------------------------------------------------------------------

[0]PETSC ERROR: Caught signal number 8 FPE: Floating Point
Exception,probably divide by zero

[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger

[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
to find memory corruption errors

[0]PETSC ERROR: likely location of problem given in stack below

[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------

[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,

[0]PETSC ERROR:       INSTEAD the line number of the start of the function

[0]PETSC ERROR:       is given.

[0]PETSC ERROR: [0] EPSGetConverged line 233
/home/soumya/slepc-3.6.0/src/eps/interface/epssolve.c

[0]PETSC ERROR: [0] EPSErrorView line 447
/home/soumya/slepc-3.6.0/src/eps/interface/epsview.c

[0]PETSC ERROR: [0] MatLUFactorSymbolic_SeqAIJ line 295
/home/soumya/petsc-3.6.1/src/mat/impls/aij/seq/aijfact.c

[0]PETSC ERROR: [0] MatLUFactorSymbolic line 2877
/home/soumya/petsc-3.6.1/src/mat/interface/matrix.c

[0]PETSC ERROR: [0] PCSetUp_LU line 99
/home/soumya/petsc-3.6.1/src/ksp/pc/impls/factor/lu/lu.c

[0]PETSC ERROR: [0] PCSetUp line 944
/home/soumya/petsc-3.6.1/src/ksp/pc/interface/precon.c

[0]PETSC ERROR: [0] KSPSetUp line 247
/home/soumya/petsc-3.6.1/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: [0] STSetUp_Cayley line 161
/home/soumya/slepc-3.6.0/src/sys/classes/st/impls/cayley/cayley.c

[0]PETSC ERROR: [0] STSetUp line 273
/home/soumya/slepc-3.6.0/src/sys/classes/st/interface/stsolve.c

[0]PETSC ERROR: [0] EPSSetUp line 58
/home/soumya/slepc-3.6.0/src/eps/interface/epssetup.c

[0]PETSC ERROR: [0] EPSSolve line 83
/home/soumya/slepc-3.6.0/src/eps/interface/epssolve.c

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------

[0]PETSC ERROR: Signal received

[0]PETSC ERROR: Seehttp://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.

[0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015

[0]PETSC ERROR: ./main on a arch-linux2-cxx-debug named
soumya-OptiPlex-9010 by soumya Mon Sep 21 21:36:29 2015

[0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran
--with-cxx=g++ --with-clanguage=cxx --download-fblaslapack --download-mpich
--with-scalar-type=complex

[0]PETSC ERROR: #12 User provided function() line 0 in  unknown file

application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

[unset]: aborting job:

application called MPI_Abort(MPI_COMM

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150921/8a02172d/attachment.html>

From jroman at dsic.upv.es  Tue Sep 22 02:49:53 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 22 Sep 2015 09:49:53 +0200
Subject: [petsc-users] PETSC error: Caught signal number 8 FPE
In-Reply-To: <CAPuCtQFUhurjiQisYOketmvwmG0UONNBPA1711oEBfLWisp0gQ@mail.gmail.com>
References: <CAPuCtQFUhurjiQisYOketmvwmG0UONNBPA1711oEBfLWisp0gQ@mail.gmail.com>
Message-ID: <44C0070A-C473-4218-85E1-A4451218C250@dsic.upv.es>

Does it happen also for STSINVERT instead of STCAYLEY?
If you provide us with a code and input data that reproduces the problem we can debug it.
Jose


> El 22/9/2015, a las 5:59, Soumya Mukherjee <soumyamechanics at gmail.com> escribi?:
> 
> 
> Hi.
> 
> I am trying to solve an eigenvalue problem. I have a code that works for a particular input file. And, whenever I change the input file, the following error message appears. Any hint for debugging, please?
> 
> PETSC ERROR: ------------------------------------------------------------------------
> 
> [0]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
> 
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> 
> [0]PETSC ERROR: or seehttp://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> 
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> 
> [0]PETSC ERROR: likely location of problem given in stack below
> 
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> 
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> 
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> 
> [0]PETSC ERROR:       is given.
> 
> [0]PETSC ERROR: [0] EPSGetConverged line 233 /home/soumya/slepc-3.6.0/src/eps/interface/epssolve.c
> 
> [0]PETSC ERROR: [0] EPSErrorView line 447 /home/soumya/slepc-3.6.0/src/eps/interface/epsview.c
> 
> [0]PETSC ERROR: [0] MatLUFactorSymbolic_SeqAIJ line 295 /home/soumya/petsc-3.6.1/src/mat/impls/aij/seq/aijfact.c
> 
> [0]PETSC ERROR: [0] MatLUFactorSymbolic line 2877 /home/soumya/petsc-3.6.1/src/mat/interface/matrix.c
> 
> [0]PETSC ERROR: [0] PCSetUp_LU line 99 /home/soumya/petsc-3.6.1/src/ksp/pc/impls/factor/lu/lu.c
> 
> [0]PETSC ERROR: [0] PCSetUp line 944 /home/soumya/petsc-3.6.1/src/ksp/pc/interface/precon.c
> 
> [0]PETSC ERROR: [0] KSPSetUp line 247 /home/soumya/petsc-3.6.1/src/ksp/ksp/interface/itfunc.c
> 
> [0]PETSC ERROR: [0] STSetUp_Cayley line 161 /home/soumya/slepc-3.6.0/src/sys/classes/st/impls/cayley/cayley.c
> 
> [0]PETSC ERROR: [0] STSetUp line 273 /home/soumya/slepc-3.6.0/src/sys/classes/st/interface/stsolve.c
> 
> [0]PETSC ERROR: [0] EPSSetUp line 58 /home/soumya/slepc-3.6.0/src/eps/interface/epssetup.c
> 
> [0]PETSC ERROR: [0] EPSSolve line 83 /home/soumya/slepc-3.6.0/src/eps/interface/epssolve.c
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> 
> [0]PETSC ERROR: Signal received
> 
> [0]PETSC ERROR: Seehttp://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> 
> [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 
> 
> [0]PETSC ERROR: ./main on a arch-linux2-cxx-debug named soumya-OptiPlex-9010 by soumya Mon Sep 21 21:36:29 2015
> 
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-fblaslapack --download-mpich --with-scalar-type=complex
> 
> [0]PETSC ERROR: #12 User provided function() line 0 in  unknown file
> 
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> 
> [unset]: aborting job:
> 
> application called MPI_Abort(MPI_COMM
> 
> Thanks,
> 


From fdkong.jd at gmail.com  Tue Sep 22 11:47:44 2015
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 22 Sep 2015 10:47:44 -0600
Subject: [petsc-users] how many downloads we have for petsc per year?
Message-ID: <CAN5Wd-J52CnDaK9EUJEC=oH+JSwYs9_7T77f=CPTVjxBUHEamw@mail.gmail.com>

Hi all,

I just was wondering how many downloads we have for petsc per year? And how
many citations from simulations based on petsc? I know the petsc is very
popular, and I have been using it for more than five years. These numbers
possibly could tell us how popular it is.


Thanks,

Fande Kong,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150922/e0d7c0a1/attachment.html>

From balay at mcs.anl.gov  Tue Sep 22 16:22:59 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 22 Sep 2015 16:22:59 -0500
Subject: [petsc-users] how many downloads we have for petsc per year?
In-Reply-To: <CAN5Wd-J52CnDaK9EUJEC=oH+JSwYs9_7T77f=CPTVjxBUHEamw@mail.gmail.com>
References: <CAN5Wd-J52CnDaK9EUJEC=oH+JSwYs9_7T77f=CPTVjxBUHEamw@mail.gmail.com>
Message-ID: <alpine.LFD.2.20.1509221606480.13989@asterix>

On Tue, 22 Sep 2015, Fande Kong wrote:

> Hi all,
> 
> I just was wondering how many downloads we have for petsc per year? And how
> many citations from simulations based on petsc? I know the petsc is very
> popular, and I have been using it for more than five years. These numbers
> possibly could tell us how popular it is.

download counts can be a bit misleading as more folk migrate to using
petsc via git. And there can be many repeat downloads - perhaps from
automated sctipts [similar to petsc nightly builds]

I have the following numbers for 2014 [and for 2015 till date]

Satish

----------
2014:
Total downloads: 191849
Unique downloads: 28237 [weeding out duplicate downloads from the same ip-address]

2015 [till-today]
Total downloads: 141673
Unique downloads: 16201

Satish

From zonexo at gmail.com  Wed Sep 23 03:14:12 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Wed, 23 Sep 2015 16:14:12 +0800
Subject: [petsc-users] Insert values into matrix using MatSetValuesStencil
In-Reply-To: <CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
References: <CADSzT4JLACgt_UZvAG+uyDkNaSOzs1t4QAr0T4DozWN--8Fe+w@mail.gmail.com>
	<CAGi1ndQLKN_+cUBchLD8SAqDn8QO46TQf+L69rjqqCLkXaX1UA@mail.gmail.com>
	<55DD3CC1.5070801@gmail.com>
	<CAGi1ndR78hB8M9_ybho-favYfdJnsissE_A45V1hFYZTQ0AC0w@mail.gmail.com>
	<55DD4869.2000006@gmail.com>
	<CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
Message-ID: <56025F54.3040604@gmail.com>

Hi,

I have successfully used MatSetValuesStencil to insert values into a 
Poisson eqn matrix which has 1 DOF (pressure). Now I'm trying to insert 
values in a momentum eqn matrix which has 3 DOF (u,v,w)

However, I got the error:

/*[0]PETSC ERROR: --------------------- Error Message 
----------------------------*//*
*//*----------------------------------*//*
*//*[0]PETSC ERROR: Argument out of range*//*
*//*[0]PETSC ERROR: Inserting a new nonzero at (111,5) in the matrix*//*
*//*[0]PETSC ERROR: See 
http://www.mcs.anl.gov/petsc/documentation/faq.html for trou*//*
*//*ble shooting.*/

I wonder what's wrong.  For the momentum eqn, for each DOF, at at node 
(dof,i,j,k), I have coupling i +/- 1, j +/- 1 and k +/- 1.

The error happens at 111,5, which corresponds to i = 2, j = 2, k = 2, 
which is an internal node.

Here's part of my code below. Hope someone can help. Thanks!
*/
/**/PetscScalar :: value_insert(7)/**/
/**/
/**/MatStencil  :: row(6,1),col(6,7)/**/
/**/
/**/ione = 1;   iseven = 7/**/
/**/
/**/if (cell_type == 'u') then/**/
/**/
/**/    offset = 1/**/
/**//**/
/**/else if (cell_type == 'v') then/**/
/**/
/**/    offset = 2/**/
/**//**/
/**/else if (cell_type == 'w') then/**/
/**/
/**/    offset = 3/**/
/**//**/
/**/end if/**/
/**/
/**/do k=ksta2,kend2/**/
/**/
/**/    do j = jsta2,jend2/**/
/**/
/**/        do i=2,size_x-1/**/
/**//**/
/**/            row(MatStencil_i,1) = i - 1/**/
/**//**/
/**/            row(MatStencil_j,1) = j - 1/**/
/**//**/
/**/            row(MatStencil_k,1) = k - 1/**/
/**//**/
/**/            row(MatStencil_c,1) = offset - 1/**/
/**//**/
/**/            value_insert = 0.d0/**/
/**//**/
/**/            col(MatStencil_i,3) = i + 1 - 1 !east/**/
/**//**/
/**/            col(MatStencil_j,3) = j - 1/**/
/**//**/
/**/            col(MatStencil_k,3) = k - 1/**/
/**//**/
/**/            col(MatStencil_c,3) = offset - 1/**/
/**//**/
/**/            value_insert(3) = -( 
1./(cell_x(i)%pd_E+cell_x(i+1)%pd_W))*(c_yz(j,k)%fc_E)*inv_Re/**/
/**//**/
/**/            col(MatStencil_i,5) = i - 1 - 1 !west/**/
/**//**/
/**/            col(MatStencil_j,5) = j - 1/**/
/**//**/
/**/            col(MatStencil_k,5) = k - 1/**/
/**//**/
/**/            col(MatStencil_c,5) = offset - 1/**/
/**//**/
/**/            value_insert(5) = -( 
1./(cell_x(i)%pd_W+cell_x(i-1)%pd_E))*(c_yz(j,k)%fc_E)*inv_Re/**/
/**//**/
/**/            col(MatStencil_i,2) = i - 1 !north/**/
/**//**/
/**/            col(MatStencil_j,2) = j + 1 - 1/**/
/**//**/
/**/            col(MatStencil_k,2) = k - 1/**/
/**//**/
/**/            col(MatStencil_c,2) = offset - 1/**/
/**//**/
/**/            value_insert(2) = -( 
1./(cell_y(j)%pd_N+cell_y(j+1)%pd_S))*(c_zx(i,k)%fc_N)*inv_Re/**/
/**//**/
/**/            col(MatStencil_i,4) = i - 1 !south/**/
/**//**/
/**/            col(MatStencil_j,4) = j - 1 - 1/**/
/**//**/
/**/            col(MatStencil_k,4) = k - 1/**/
/**//**/
/**/            col(MatStencil_c,4) = offset - 1/**/
/**//**/
/**/            value_insert(4) = -( 
1./(cell_y(j)%pd_S+cell_y(j-1)%pd_N))*(c_zx(i,k)%fc_N)*inv_Re/**/
/**//**/
/**/            col(MatStencil_i,6) = i - 1 !front/**/
/**//**/
/**/            col(MatStencil_j,6) = j - 1/**/
/**//**/
/**/            col(MatStencil_k,6) = k + 1 - 1/**/
/**//**/
/**/            col(MatStencil_c,6) = offset - 1/**/
/**//**/
/**/            value_insert(6) = -( 
1./(cell_z(k)%pd_F+cell_z(k+1)%pd_B))*(c_xy(i,j)%fc_F)*inv_Re/**/
/**//**/
/**/            col(MatStencil_i,7) = i - 1 !back/**/
/**//**/
/**/            col(MatStencil_j,7) = j - 1/**/
/**//**/
/**/            col(MatStencil_k,7) = k - 1 - 1/**/
/**//**/
/**/            col(MatStencil_c,7) = offset - 1/**/
/**//**/
/**/            value_insert(7) = -( 
1./(cell_z(k)%pd_B+cell_z(k-1)%pd_F))*(c_xy(i,j)%fc_F)*inv_Re/**/
/**//**/
/**/            col(MatStencil_i,1) = i - 1/**/
/**//**/
/**/            col(MatStencil_j,1) = j - 1/**/
/**//**/
/**/            col(MatStencil_k,1) = k - 1/**/
/**//**/
/**/            col(MatStencil_c,1) = offset - 1/**/
/**//**/
/**/            value_insert(1) = 2.*c(i,j,k)%vol/del_t - 
(value_insert(2)+value_insert(3)+value_insert(4)+value_insert(5)+value_insert(6)+value_insert(7))/**/
/**//**/
/**/            call 
MatSetValuesStencil(A_semi_xyz,ione,row,iseven,col,value_insert,INSERT_VALUES,ierr)/**/
/**//**/
/**/        end do/**/
/**//**/
/**/    end do/**/
/**/
/**/end do /*

Thank you

Yours sincerely,

TAY wee-beng


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150923/4a386743/attachment.html>

From timothee.nicolas at gmail.com  Wed Sep 23 03:18:11 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Wed, 23 Sep 2015 17:18:11 +0900
Subject: [petsc-users] Insert values into matrix using
	MatSetValuesStencil
In-Reply-To: <56025F54.3040604@gmail.com>
References: <CADSzT4JLACgt_UZvAG+uyDkNaSOzs1t4QAr0T4DozWN--8Fe+w@mail.gmail.com>
	<CAGi1ndQLKN_+cUBchLD8SAqDn8QO46TQf+L69rjqqCLkXaX1UA@mail.gmail.com>
	<55DD3CC1.5070801@gmail.com>
	<CAGi1ndR78hB8M9_ybho-favYfdJnsissE_A45V1hFYZTQ0AC0w@mail.gmail.com>
	<55DD4869.2000006@gmail.com>
	<CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
	<56025F54.3040604@gmail.com>
Message-ID: <CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>

The first thing that strikes me is your definition of the stencils

*MatStencil  :: row(6,1),col(6,7)*

Why is it not defined with

*MatStencil  :: row(4,1),col(4,7)*

instead ?

Where does the 6 come from ?

Timoth?e


2015-09-23 17:14 GMT+09:00 TAY wee-beng <zonexo at gmail.com>:

> Hi,
>
> I have successfully used MatSetValuesStencil to insert values into a
> Poisson eqn matrix which has 1 DOF (pressure). Now I'm trying to insert
> values in a momentum eqn matrix which has 3 DOF (u,v,w)
>
> However, I got the error:
>
> *[0]PETSC ERROR: --------------------- Error Message
> ----------------------------*
> *----------------------------------*
> *[0]PETSC ERROR: Argument out of range*
> *[0]PETSC ERROR: Inserting a new nonzero at (111,5) in the matrix*
> *[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trou*
> *ble shooting.*
>
> I wonder what's wrong.  For the momentum eqn, for each DOF, at at node
> (dof,i,j,k), I have coupling i +/- 1, j +/- 1 and k +/- 1.
>
> The error happens at 111,5, which corresponds to i = 2, j = 2, k = 2,
> which is an internal node.
>
> Here's part of my code below. Hope someone can help. Thanks!
>
> *PetscScalar :: value_insert(7)*
>
> *MatStencil  :: row(6,1),col(6,7)*
>
> *ione = 1;   iseven = 7*
>
> *if (cell_type == 'u') then*
>
> *    offset = 1*
>
> *else if (cell_type == 'v') then*
>
> *    offset = 2*
>
> *else if (cell_type == 'w') then*
>
> *    offset = 3*
>
> *end if*
>
> *do k=ksta2,kend2*
>
> *    do j = jsta2,jend2*
>
> *        do i=2,size_x-1*
>
> *            row(MatStencil_i,1) = i - 1*
>
> *            row(MatStencil_j,1) = j - 1*
>
> *            row(MatStencil_k,1) = k - 1*
>
> *            row(MatStencil_c,1) = offset - 1*
>
> *            value_insert = 0.d0*
>
> *            col(MatStencil_i,3) = i + 1 - 1 !east*
>
> *            col(MatStencil_j,3) = j - 1*
>
> *            col(MatStencil_k,3) = k - 1*
>
> *            col(MatStencil_c,3) = offset - 1*
>
> *            value_insert(3) = -(
> 1./(cell_x(i)%pd_E+cell_x(i+1)%pd_W))*(c_yz(j,k)%fc_E)*inv_Re*
>
> *            col(MatStencil_i,5) = i - 1 - 1 !west*
>
> *            col(MatStencil_j,5) = j - 1*
>
> *            col(MatStencil_k,5) = k - 1*
>
> *            col(MatStencil_c,5) = offset - 1*
>
> *            value_insert(5) = -(
> 1./(cell_x(i)%pd_W+cell_x(i-1)%pd_E))*(c_yz(j,k)%fc_E)*inv_Re*
>
> *            col(MatStencil_i,2) = i - 1 !north*
>
> *            col(MatStencil_j,2) = j + 1 - 1*
>
> *            col(MatStencil_k,2) = k - 1*
>
> *            col(MatStencil_c,2) = offset - 1*
>
> *            value_insert(2) = -(
> 1./(cell_y(j)%pd_N+cell_y(j+1)%pd_S))*(c_zx(i,k)%fc_N)*inv_Re*
>
> *            col(MatStencil_i,4) = i - 1 !south*
>
> *            col(MatStencil_j,4) = j - 1 - 1*
>
> *            col(MatStencil_k,4) = k - 1*
>
> *            col(MatStencil_c,4) = offset - 1*
>
> *            value_insert(4) = -(
> 1./(cell_y(j)%pd_S+cell_y(j-1)%pd_N))*(c_zx(i,k)%fc_N)*inv_Re*
>
> *            col(MatStencil_i,6) = i - 1 !front*
>
> *            col(MatStencil_j,6) = j - 1*
>
> *            col(MatStencil_k,6) = k + 1 - 1*
>
> *            col(MatStencil_c,6) = offset - 1*
>
> *            value_insert(6) = -(
> 1./(cell_z(k)%pd_F+cell_z(k+1)%pd_B))*(c_xy(i,j)%fc_F)*inv_Re*
>
> *            col(MatStencil_i,7) = i - 1 !back*
>
> *            col(MatStencil_j,7) = j - 1*
>
> *            col(MatStencil_k,7) = k - 1 - 1*
>
> *            col(MatStencil_c,7) = offset - 1*
>
> *            value_insert(7) = -(
> 1./(cell_z(k)%pd_B+cell_z(k-1)%pd_F))*(c_xy(i,j)%fc_F)*inv_Re*
>
> *            col(MatStencil_i,1) = i - 1*
>
> *            col(MatStencil_j,1) = j - 1*
>
> *            col(MatStencil_k,1) = k - 1*
>
> *            col(MatStencil_c,1) = offset - 1*
>
> *            value_insert(1) = 2.*c(i,j,k)%vol/del_t -
> (value_insert(2)+value_insert(3)+value_insert(4)+value_insert(5)+value_insert(6)+value_insert(7))*
>
> *            call
> MatSetValuesStencil(A_semi_xyz,ione,row,iseven,col,value_insert,INSERT_VALUES,ierr)*
>
> *        end do*
>
> *    end do*
>
> *end do    *
>
> Thank you
>
> Yours sincerely,
>
> TAY wee-beng
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150923/18a95e76/attachment-0001.html>

From timothee.nicolas at gmail.com  Wed Sep 23 03:22:27 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Wed, 23 Sep 2015 17:22:27 +0900
Subject: [petsc-users] Insert values into matrix using
	MatSetValuesStencil
In-Reply-To: <CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>
References: <CADSzT4JLACgt_UZvAG+uyDkNaSOzs1t4QAr0T4DozWN--8Fe+w@mail.gmail.com>
	<CAGi1ndQLKN_+cUBchLD8SAqDn8QO46TQf+L69rjqqCLkXaX1UA@mail.gmail.com>
	<55DD3CC1.5070801@gmail.com>
	<CAGi1ndR78hB8M9_ybho-favYfdJnsissE_A45V1hFYZTQ0AC0w@mail.gmail.com>
	<55DD4869.2000006@gmail.com>
	<CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
	<56025F54.3040604@gmail.com>
	<CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>
Message-ID: <CAGi1ndRp69mEq8VnK86Uc8HYvZ6b5zXjYdKvtqkw3niC2ew9-Q@mail.gmail.com>

Can you also tell how you created the matrix ? Just in case you created it
with the 1 dof DMDA, it would not work if you try to input values at places
where it is not allocated (which could explain the error message)

2015-09-23 17:18 GMT+09:00 Timoth?e Nicolas <timothee.nicolas at gmail.com>:

> The first thing that strikes me is your definition of the stencils
>
> *MatStencil  :: row(6,1),col(6,7)*
>
> Why is it not defined with
>
> *MatStencil  :: row(4,1),col(4,7)*
>
> instead ?
>
> Where does the 6 come from ?
>
> Timoth?e
>
>
> 2015-09-23 17:14 GMT+09:00 TAY wee-beng <zonexo at gmail.com>:
>
>> Hi,
>>
>> I have successfully used MatSetValuesStencil to insert values into a
>> Poisson eqn matrix which has 1 DOF (pressure). Now I'm trying to insert
>> values in a momentum eqn matrix which has 3 DOF (u,v,w)
>>
>> However, I got the error:
>>
>> *[0]PETSC ERROR: --------------------- Error Message
>> ----------------------------*
>> *----------------------------------*
>> *[0]PETSC ERROR: Argument out of range*
>> *[0]PETSC ERROR: Inserting a new nonzero at (111,5) in the matrix*
>> *[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trou*
>> *ble shooting.*
>>
>> I wonder what's wrong.  For the momentum eqn, for each DOF, at at node
>> (dof,i,j,k), I have coupling i +/- 1, j +/- 1 and k +/- 1.
>>
>> The error happens at 111,5, which corresponds to i = 2, j = 2, k = 2,
>> which is an internal node.
>>
>> Here's part of my code below. Hope someone can help. Thanks!
>>
>> *PetscScalar :: value_insert(7)*
>>
>> *MatStencil  :: row(6,1),col(6,7)*
>>
>> *ione = 1;   iseven = 7*
>>
>> *if (cell_type == 'u') then*
>>
>> *    offset = 1*
>>
>> *else if (cell_type == 'v') then*
>>
>> *    offset = 2*
>>
>> *else if (cell_type == 'w') then*
>>
>> *    offset = 3*
>>
>> *end if*
>>
>> *do k=ksta2,kend2*
>>
>> *    do j = jsta2,jend2*
>>
>> *        do i=2,size_x-1*
>>
>> *            row(MatStencil_i,1) = i - 1*
>>
>> *            row(MatStencil_j,1) = j - 1*
>>
>> *            row(MatStencil_k,1) = k - 1*
>>
>> *            row(MatStencil_c,1) = offset - 1*
>>
>> *            value_insert = 0.d0*
>>
>> *            col(MatStencil_i,3) = i + 1 - 1 !east*
>>
>> *            col(MatStencil_j,3) = j - 1*
>>
>> *            col(MatStencil_k,3) = k - 1*
>>
>> *            col(MatStencil_c,3) = offset - 1*
>>
>> *            value_insert(3) = -(
>> 1./(cell_x(i)%pd_E+cell_x(i+1)%pd_W))*(c_yz(j,k)%fc_E)*inv_Re*
>>
>> *            col(MatStencil_i,5) = i - 1 - 1 !west*
>>
>> *            col(MatStencil_j,5) = j - 1*
>>
>> *            col(MatStencil_k,5) = k - 1*
>>
>> *            col(MatStencil_c,5) = offset - 1*
>>
>> *            value_insert(5) = -(
>> 1./(cell_x(i)%pd_W+cell_x(i-1)%pd_E))*(c_yz(j,k)%fc_E)*inv_Re*
>>
>> *            col(MatStencil_i,2) = i - 1 !north*
>>
>> *            col(MatStencil_j,2) = j + 1 - 1*
>>
>> *            col(MatStencil_k,2) = k - 1*
>>
>> *            col(MatStencil_c,2) = offset - 1*
>>
>> *            value_insert(2) = -(
>> 1./(cell_y(j)%pd_N+cell_y(j+1)%pd_S))*(c_zx(i,k)%fc_N)*inv_Re*
>>
>> *            col(MatStencil_i,4) = i - 1 !south*
>>
>> *            col(MatStencil_j,4) = j - 1 - 1*
>>
>> *            col(MatStencil_k,4) = k - 1*
>>
>> *            col(MatStencil_c,4) = offset - 1*
>>
>> *            value_insert(4) = -(
>> 1./(cell_y(j)%pd_S+cell_y(j-1)%pd_N))*(c_zx(i,k)%fc_N)*inv_Re*
>>
>> *            col(MatStencil_i,6) = i - 1 !front*
>>
>> *            col(MatStencil_j,6) = j - 1*
>>
>> *            col(MatStencil_k,6) = k + 1 - 1*
>>
>> *            col(MatStencil_c,6) = offset - 1*
>>
>> *            value_insert(6) = -(
>> 1./(cell_z(k)%pd_F+cell_z(k+1)%pd_B))*(c_xy(i,j)%fc_F)*inv_Re*
>>
>> *            col(MatStencil_i,7) = i - 1 !back*
>>
>> *            col(MatStencil_j,7) = j - 1*
>>
>> *            col(MatStencil_k,7) = k - 1 - 1*
>>
>> *            col(MatStencil_c,7) = offset - 1*
>>
>> *            value_insert(7) = -(
>> 1./(cell_z(k)%pd_B+cell_z(k-1)%pd_F))*(c_xy(i,j)%fc_F)*inv_Re*
>>
>> *            col(MatStencil_i,1) = i - 1*
>>
>> *            col(MatStencil_j,1) = j - 1*
>>
>> *            col(MatStencil_k,1) = k - 1*
>>
>> *            col(MatStencil_c,1) = offset - 1*
>>
>> *            value_insert(1) = 2.*c(i,j,k)%vol/del_t -
>> (value_insert(2)+value_insert(3)+value_insert(4)+value_insert(5)+value_insert(6)+value_insert(7))*
>>
>> *            call
>> MatSetValuesStencil(A_semi_xyz,ione,row,iseven,col,value_insert,INSERT_VALUES,ierr)*
>>
>> *        end do*
>>
>> *    end do*
>>
>> *end do    *
>>
>> Thank you
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150923/1e86eaa6/attachment.html>

From zonexo at gmail.com  Wed Sep 23 03:25:28 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Wed, 23 Sep 2015 16:25:28 +0800
Subject: [petsc-users] Insert values into matrix using
	MatSetValuesStencil
In-Reply-To: <CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>
References: <CADSzT4JLACgt_UZvAG+uyDkNaSOzs1t4QAr0T4DozWN--8Fe+w@mail.gmail.com>
	<CAGi1ndQLKN_+cUBchLD8SAqDn8QO46TQf+L69rjqqCLkXaX1UA@mail.gmail.com>
	<55DD3CC1.5070801@gmail.com>
	<CAGi1ndR78hB8M9_ybho-favYfdJnsissE_A45V1hFYZTQ0AC0w@mail.gmail.com>
	<55DD4869.2000006@gmail.com>
	<CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
	<56025F54.3040604@gmail.com>
	<CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>
Message-ID: <560261F8.9090700@gmail.com>


Thank you

Yours sincerely,

TAY wee-beng

On 23/9/2015 4:18 PM, Timoth?e Nicolas wrote:
> The first thing that strikes me is your definition of the stencils
>
> */MatStencil  :: row(6,1),col(6,7)/**/
>
> /*
> Why is it not defined with
>
> */MatStencil  :: row(4,1),col(4,7)/**/
> /*
> instead ?
>
> Where does the 6 come from ?
>
> Timoth?e
Hi,

Maybe I misunderstood. But in your email tutorial, last time, with 1 
dof, it's


*/MatStencil  :: row(4,1),col(4,7)/**/
/*
So now I have 3 dof, shouldn't it be

*/MatStencil  :: row(6,1),col(6,7)/**/
?

/*
>
>
> 2015-09-23 17:14 GMT+09:00 TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>>:
>
>     Hi,
>
>     I have successfully used MatSetValuesStencil to insert values into
>     a Poisson eqn matrix which has 1 DOF (pressure). Now I'm trying to
>     insert values in a momentum eqn matrix which has 3 DOF (u,v,w)
>
>     However, I got the error:
>
>     /*[0]PETSC ERROR: --------------------- Error Message
>     ----------------------------*//*
>     *//*----------------------------------*//*
>     *//*[0]PETSC ERROR: Argument out of range*//*
>     *//*[0]PETSC ERROR: Inserting a new nonzero at (111,5) in the
>     matrix*//*
>     *//*[0]PETSC ERROR: See
>     http://www.mcs.anl.gov/petsc/documentation/faq.html for trou*//*
>     *//*ble shooting.*/
>
>     I wonder what's wrong.  For the momentum eqn, for each DOF, at at
>     node (dof,i,j,k), I have coupling i +/- 1, j +/- 1 and k +/- 1.
>
>     The error happens at 111,5, which corresponds to i = 2, j = 2, k =
>     2, which is an internal node.
>
>     Here's part of my code below. Hope someone can help. Thanks!
>     */
>     /**/PetscScalar :: value_insert(7)/**/
>     /**/
>     /**/MatStencil  :: row(6,1),col(6,7)/**/
>     /**/
>     /**/ione = 1;   iseven = 7/**/
>     /**/
>     /**/if (cell_type == 'u') then/**/
>     /**/
>     /**/    offset = 1/**/
>     /**//**/
>     /**/else if (cell_type == 'v') then/**/
>     /**/
>     /**/    offset = 2/**/
>     /**//**/
>     /**/else if (cell_type == 'w') then/**/
>     /**/
>     /**/    offset = 3/**/
>     /**//**/
>     /**/end if/**/
>     /**/
>     /**/do k=ksta2,kend2/**/
>     /**/
>     /**/    do j = jsta2,jend2/**/
>     /**/
>     /**/        do i=2,size_x-1/**/
>     /**//**/
>     /**/            row(MatStencil_i,1) = i - 1/**/
>     /**//**/
>     /**/            row(MatStencil_j,1) = j - 1/**/
>     /**//**/
>     /**/            row(MatStencil_k,1) = k - 1/**/
>     /**//**/
>     /**/            row(MatStencil_c,1) = offset - 1/**/
>     /**//**/
>     /**/            value_insert = 0.d0/**/
>     /**//**/
>     /**/            col(MatStencil_i,3) = i + 1 - 1 !east/**/
>     /**//**/
>     /**/            col(MatStencil_j,3) = j - 1/**/
>     /**//**/
>     /**/            col(MatStencil_k,3) = k - 1/**/
>     /**//**/
>     /**/            col(MatStencil_c,3) = offset - 1/**/
>     /**//**/
>     /**/            value_insert(3) = -(
>     1./(cell_x(i)%pd_E+cell_x(i+1)%pd_W))*(c_yz(j,k)%fc_E)*inv_Re/**/
>     /**//**/
>     /**/            col(MatStencil_i,5) = i - 1 - 1 !west/**/
>     /**//**/
>     /**/            col(MatStencil_j,5) = j - 1/**/
>     /**//**/
>     /**/            col(MatStencil_k,5) = k - 1/**/
>     /**//**/
>     /**/            col(MatStencil_c,5) = offset - 1/**/
>     /**//**/
>     /**/            value_insert(5) = -(
>     1./(cell_x(i)%pd_W+cell_x(i-1)%pd_E))*(c_yz(j,k)%fc_E)*inv_Re/**/
>     /**//**/
>     /**/            col(MatStencil_i,2) = i - 1 !north/**/
>     /**//**/
>     /**/            col(MatStencil_j,2) = j + 1 - 1/**/
>     /**//**/
>     /**/            col(MatStencil_k,2) = k - 1/**/
>     /**//**/
>     /**/            col(MatStencil_c,2) = offset - 1/**/
>     /**//**/
>     /**/            value_insert(2) = -(
>     1./(cell_y(j)%pd_N+cell_y(j+1)%pd_S))*(c_zx(i,k)%fc_N)*inv_Re/**/
>     /**//**/
>     /**/            col(MatStencil_i,4) = i - 1 !south/**/
>     /**//**/
>     /**/            col(MatStencil_j,4) = j - 1 - 1/**/
>     /**//**/
>     /**/            col(MatStencil_k,4) = k - 1/**/
>     /**//**/
>     /**/            col(MatStencil_c,4) = offset - 1/**/
>     /**//**/
>     /**/            value_insert(4) = -(
>     1./(cell_y(j)%pd_S+cell_y(j-1)%pd_N))*(c_zx(i,k)%fc_N)*inv_Re/**/
>     /**//**/
>     /**/            col(MatStencil_i,6) = i - 1 !front/**/
>     /**//**/
>     /**/            col(MatStencil_j,6) = j - 1/**/
>     /**//**/
>     /**/            col(MatStencil_k,6) = k + 1 - 1/**/
>     /**//**/
>     /**/            col(MatStencil_c,6) = offset - 1/**/
>     /**//**/
>     /**/            value_insert(6) = -(
>     1./(cell_z(k)%pd_F+cell_z(k+1)%pd_B))*(c_xy(i,j)%fc_F)*inv_Re/**/
>     /**//**/
>     /**/            col(MatStencil_i,7) = i - 1 !back/**/
>     /**//**/
>     /**/            col(MatStencil_j,7) = j - 1/**/
>     /**//**/
>     /**/            col(MatStencil_k,7) = k - 1 - 1/**/
>     /**//**/
>     /**/            col(MatStencil_c,7) = offset - 1/**/
>     /**//**/
>     /**/            value_insert(7) = -(
>     1./(cell_z(k)%pd_B+cell_z(k-1)%pd_F))*(c_xy(i,j)%fc_F)*inv_Re/**/
>     /**//**/
>     /**/            col(MatStencil_i,1) = i - 1/**/
>     /**//**/
>     /**/            col(MatStencil_j,1) = j - 1/**/
>     /**//**/
>     /**/            col(MatStencil_k,1) = k - 1/**/
>     /**//**/
>     /**/            col(MatStencil_c,1) = offset - 1/**/
>     /**//**/
>     /**/            value_insert(1) = 2.*c(i,j,k)%vol/del_t -
>     (value_insert(2)+value_insert(3)+value_insert(4)+value_insert(5)+value_insert(6)+value_insert(7))/**/
>     /**//**/
>     /**/            call
>     MatSetValuesStencil(A_semi_xyz,ione,row,iseven,col,value_insert,INSERT_VALUES,ierr)/**/
>     /**//**/
>     /**/        end do/**/
>     /**//**/
>     /**/    end do/**/
>     /**/
>     /**/end do /*
>
>     Thank you
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150923/4494896b/attachment-0001.html>

From timothee.nicolas at gmail.com  Wed Sep 23 03:30:49 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Wed, 23 Sep 2015 17:30:49 +0900
Subject: [petsc-users] Insert values into matrix using
	MatSetValuesStencil
In-Reply-To: <560261F8.9090700@gmail.com>
References: <CADSzT4JLACgt_UZvAG+uyDkNaSOzs1t4QAr0T4DozWN--8Fe+w@mail.gmail.com>
	<CAGi1ndQLKN_+cUBchLD8SAqDn8QO46TQf+L69rjqqCLkXaX1UA@mail.gmail.com>
	<55DD3CC1.5070801@gmail.com>
	<CAGi1ndR78hB8M9_ybho-favYfdJnsissE_A45V1hFYZTQ0AC0w@mail.gmail.com>
	<55DD4869.2000006@gmail.com>
	<CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
	<56025F54.3040604@gmail.com>
	<CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>
	<560261F8.9090700@gmail.com>
Message-ID: <CAGi1ndRm8ZVk2BcbC=GgEeJ=Z8J4kB35=rrUzHNJiA0pHsrk5w@mail.gmail.com>

No, you misunderstood. The 4 refers to

1 for i direction
+
1 for j direction
+
1 for k direction
+
1 for dof direction
=
4

Does it solve your problem ?

Best

Timothee

2015-09-23 17:25 GMT+09:00 TAY wee-beng <zonexo at gmail.com>:

>
> Thank you
>
> Yours sincerely,
>
> TAY wee-beng
>
> On 23/9/2015 4:18 PM, Timoth?e Nicolas wrote:
>
> The first thing that strikes me is your definition of the stencils
>
> *MatStencil  :: row(6,1),col(6,7)*
>
> Why is it not defined with
>
> *MatStencil  :: row(4,1),col(4,7)*
>
> instead ?
>
> Where does the 6 come from ?
>
> Timoth?e
>
> Hi,
>
> Maybe I misunderstood. But in your email tutorial, last time, with 1 dof,
> it's
>
>
> *MatStencil  :: row(4,1),col(4,7)*
>
> So now I have 3 dof, shouldn't it be
>
> *MatStencil  :: row(6,1),col(6,7)*
>
>
> * ? *
>
>
>
> 2015-09-23 17:14 GMT+09:00 TAY wee-beng <zonexo at gmail.com>:
>
>> Hi,
>>
>> I have successfully used MatSetValuesStencil to insert values into a
>> Poisson eqn matrix which has 1 DOF (pressure). Now I'm trying to insert
>> values in a momentum eqn matrix which has 3 DOF (u,v,w)
>>
>> However, I got the error:
>>
>> *[0]PETSC ERROR: --------------------- Error Message
>> ----------------------------*
>> *----------------------------------*
>> *[0]PETSC ERROR: Argument out of range*
>> *[0]PETSC ERROR: Inserting a new nonzero at (111,5) in the matrix*
>> *[0]PETSC ERROR: See
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html>http://www.mcs.anl.gov/petsc/documentation/faq.html
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trou*
>> *ble shooting.*
>>
>> I wonder what's wrong.  For the momentum eqn, for each DOF, at at node
>> (dof,i,j,k), I have coupling i +/- 1, j +/- 1 and k +/- 1.
>>
>> The error happens at 111,5, which corresponds to i = 2, j = 2, k = 2,
>> which is an internal node.
>>
>> Here's part of my code below. Hope someone can help. Thanks!
>>
>> *PetscScalar :: value_insert(7)*
>>
>> *MatStencil  :: row(6,1),col(6,7)*
>>
>> *ione = 1;   iseven = 7*
>>
>> *if (cell_type == 'u') then*
>>
>> *    offset = 1*
>>
>> *else if (cell_type == 'v') then*
>>
>> *    offset = 2*
>>
>> *else if (cell_type == 'w') then*
>>
>> *    offset = 3*
>>
>> *end if*
>>
>> *do k=ksta2,kend2*
>>
>> *    do j = jsta2,jend2*
>>
>> *        do i=2,size_x-1*
>>
>> *            row(MatStencil_i,1) = i - 1*
>>
>> *            row(MatStencil_j,1) = j - 1*
>>
>> *            row(MatStencil_k,1) = k - 1*
>>
>> *            row(MatStencil_c,1) = offset - 1*
>>
>> *            value_insert = 0.d0*
>>
>> *            col(MatStencil_i,3) = i + 1 - 1 !east*
>>
>> *            col(MatStencil_j,3) = j - 1*
>>
>> *            col(MatStencil_k,3) = k - 1*
>>
>> *            col(MatStencil_c,3) = offset - 1*
>>
>> *            value_insert(3) = -(
>> 1./(cell_x(i)%pd_E+cell_x(i+1)%pd_W))*(c_yz(j,k)%fc_E)*inv_Re*
>>
>> *            col(MatStencil_i,5) = i - 1 - 1 !west*
>>
>> *            col(MatStencil_j,5) = j - 1*
>>
>> *            col(MatStencil_k,5) = k - 1*
>>
>> *            col(MatStencil_c,5) = offset - 1*
>>
>> *            value_insert(5) = -(
>> 1./(cell_x(i)%pd_W+cell_x(i-1)%pd_E))*(c_yz(j,k)%fc_E)*inv_Re*
>>
>> *            col(MatStencil_i,2) = i - 1 !north*
>>
>> *            col(MatStencil_j,2) = j + 1 - 1*
>>
>> *            col(MatStencil_k,2) = k - 1*
>>
>> *            col(MatStencil_c,2) = offset - 1*
>>
>> *            value_insert(2) = -(
>> 1./(cell_y(j)%pd_N+cell_y(j+1)%pd_S))*(c_zx(i,k)%fc_N)*inv_Re*
>>
>> *            col(MatStencil_i,4) = i - 1 !south*
>>
>> *            col(MatStencil_j,4) = j - 1 - 1*
>>
>> *            col(MatStencil_k,4) = k - 1*
>>
>> *            col(MatStencil_c,4) = offset - 1*
>>
>> *            value_insert(4) = -(
>> 1./(cell_y(j)%pd_S+cell_y(j-1)%pd_N))*(c_zx(i,k)%fc_N)*inv_Re*
>>
>> *            col(MatStencil_i,6) = i - 1 !front*
>>
>> *            col(MatStencil_j,6) = j - 1*
>>
>> *            col(MatStencil_k,6) = k + 1 - 1*
>>
>> *            col(MatStencil_c,6) = offset - 1*
>>
>> *            value_insert(6) = -(
>> 1./(cell_z(k)%pd_F+cell_z(k+1)%pd_B))*(c_xy(i,j)%fc_F)*inv_Re*
>>
>> *            col(MatStencil_i,7) = i - 1 !back*
>>
>> *            col(MatStencil_j,7) = j - 1*
>>
>> *            col(MatStencil_k,7) = k - 1 - 1*
>>
>> *            col(MatStencil_c,7) = offset - 1*
>>
>> *            value_insert(7) = -(
>> 1./(cell_z(k)%pd_B+cell_z(k-1)%pd_F))*(c_xy(i,j)%fc_F)*inv_Re*
>>
>> *            col(MatStencil_i,1) = i - 1*
>>
>> *            col(MatStencil_j,1) = j - 1*
>>
>> *            col(MatStencil_k,1) = k - 1*
>>
>> *            col(MatStencil_c,1) = offset - 1*
>>
>> *            value_insert(1) = 2.*c(i,j,k)%vol/del_t -
>> (value_insert(2)+value_insert(3)+value_insert(4)+value_insert(5)+value_insert(6)+value_insert(7))*
>>
>> *            call
>> MatSetValuesStencil(A_semi_xyz,ione,row,iseven,col,value_insert,INSERT_VALUES,ierr)*
>>
>> *        end do*
>>
>> *    end do*
>>
>> *end do    *
>>
>> Thank you
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150923/bd520e9d/attachment.html>

From zonexo at gmail.com  Wed Sep 23 03:37:24 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Wed, 23 Sep 2015 16:37:24 +0800
Subject: [petsc-users] Insert values into matrix using
	MatSetValuesStencil
In-Reply-To: <CAGi1ndRp69mEq8VnK86Uc8HYvZ6b5zXjYdKvtqkw3niC2ew9-Q@mail.gmail.com>
References: <CADSzT4JLACgt_UZvAG+uyDkNaSOzs1t4QAr0T4DozWN--8Fe+w@mail.gmail.com>
	<CAGi1ndQLKN_+cUBchLD8SAqDn8QO46TQf+L69rjqqCLkXaX1UA@mail.gmail.com>
	<55DD3CC1.5070801@gmail.com>
	<CAGi1ndR78hB8M9_ybho-favYfdJnsissE_A45V1hFYZTQ0AC0w@mail.gmail.com>
	<55DD4869.2000006@gmail.com>
	<CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
	<56025F54.3040604@gmail.com>
	<CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>
	<CAGi1ndRp69mEq8VnK86Uc8HYvZ6b5zXjYdKvtqkw3niC2ew9-Q@mail.gmail.com>
Message-ID: <560264C4.70205@gmail.com>

Hi Timoth?e,

The matrix is created with 3 dof - u,v,w. So for each i,j,k, there are 3 
values. Actually I got 3 eqns from the u,v,w momentum eqns. They are not 
coupled together so I can also solve them individually. But I was told 
it's faster to group them together. The global indices is something like 
this:

i, j, k, ijk global indices, dof
1 1 1 1 u
1 1 1 2 v
1 1 1 3 w
2 1 1 4 u

...

Hope it's clearer now.

Ok, I changed the stencil, it's now working.

Thank you

Yours sincerely,

TAY wee-beng

On 23/9/2015 4:22 PM, Timoth?e Nicolas wrote:
> Can you also tell how you created the matrix ? Just in case you 
> created it with the 1 dof DMDA, it would not work if you try to input 
> values at places where it is not allocated (which could explain the 
> error message)
>
> 2015-09-23 17:18 GMT+09:00 Timoth?e Nicolas 
> <timothee.nicolas at gmail.com <mailto:timothee.nicolas at gmail.com>>:
>
>     The first thing that strikes me is your definition of the stencils
>
>     */MatStencil  :: row(6,1),col(6,7)/**/
>
>     /*
>     Why is it not defined with
>
>     */MatStencil  :: row(4,1),col(4,7)/**/
>     /*
>     instead ?
>
>     Where does the 6 come from ?
>
>     Timoth?e
>
>
>     2015-09-23 17:14 GMT+09:00 TAY wee-beng <zonexo at gmail.com
>     <mailto:zonexo at gmail.com>>:
>
>         Hi,
>
>         I have successfully used MatSetValuesStencil to insert values
>         into a Poisson eqn matrix which has 1 DOF (pressure). Now I'm
>         trying to insert values in a momentum eqn matrix which has 3
>         DOF (u,v,w)
>
>         However, I got the error:
>
>         /*[0]PETSC ERROR: --------------------- Error Message
>         ----------------------------*//*
>         *//*----------------------------------*//*
>         *//*[0]PETSC ERROR: Argument out of range*//*
>         *//*[0]PETSC ERROR: Inserting a new nonzero at (111,5) in the
>         matrix*//*
>         *//*[0]PETSC ERROR: See
>         http://www.mcs.anl.gov/petsc/documentation/faq.html for trou*//*
>         *//*ble shooting.*/
>
>         I wonder what's wrong.  For the momentum eqn, for each DOF, at
>         at node (dof,i,j,k), I have coupling i +/- 1, j +/- 1 and k +/- 1.
>
>         The error happens at 111,5, which corresponds to i = 2, j = 2,
>         k = 2, which is an internal node.
>
>         Here's part of my code below. Hope someone can help. Thanks!
>         */
>         /**/PetscScalar :: value_insert(7)/**/
>         /**/
>         /**/MatStencil  :: row(6,1),col(6,7)/**/
>         /**/
>         /**/ione = 1;   iseven = 7/**/
>         /**/
>         /**/if (cell_type == 'u') then/**/
>         /**/
>         /**/    offset = 1/**/
>         /**//**/
>         /**/else if (cell_type == 'v') then/**/
>         /**/
>         /**/    offset = 2/**/
>         /**//**/
>         /**/else if (cell_type == 'w') then/**/
>         /**/
>         /**/    offset = 3/**/
>         /**//**/
>         /**/end if/**/
>         /**/
>         /**/do k=ksta2,kend2/**/
>         /**/
>         /**/    do j = jsta2,jend2/**/
>         /**/
>         /**/        do i=2,size_x-1/**/
>         /**//**/
>         /**/            row(MatStencil_i,1) = i - 1/**/
>         /**//**/
>         /**/            row(MatStencil_j,1) = j - 1/**/
>         /**//**/
>         /**/            row(MatStencil_k,1) = k - 1/**/
>         /**//**/
>         /**/            row(MatStencil_c,1) = offset - 1/**/
>         /**//**/
>         /**/            value_insert = 0.d0/**/
>         /**//**/
>         /**/            col(MatStencil_i,3) = i + 1 - 1 !east/**/
>         /**//**/
>         /**/            col(MatStencil_j,3) = j - 1/**/
>         /**//**/
>         /**/            col(MatStencil_k,3) = k - 1/**/
>         /**//**/
>         /**/            col(MatStencil_c,3) = offset - 1/**/
>         /**//**/
>         /**/            value_insert(3) = -(
>         1./(cell_x(i)%pd_E+cell_x(i+1)%pd_W))*(c_yz(j,k)%fc_E)*inv_Re/**/
>         /**//**/
>         /**/            col(MatStencil_i,5) = i - 1 - 1 !west/**/
>         /**//**/
>         /**/            col(MatStencil_j,5) = j - 1/**/
>         /**//**/
>         /**/            col(MatStencil_k,5) = k - 1/**/
>         /**//**/
>         /**/            col(MatStencil_c,5) = offset - 1/**/
>         /**//**/
>         /**/            value_insert(5) = -(
>         1./(cell_x(i)%pd_W+cell_x(i-1)%pd_E))*(c_yz(j,k)%fc_E)*inv_Re/**/
>         /**//**/
>         /**/            col(MatStencil_i,2) = i - 1 !north/**/
>         /**//**/
>         /**/            col(MatStencil_j,2) = j + 1 - 1/**/
>         /**//**/
>         /**/            col(MatStencil_k,2) = k - 1/**/
>         /**//**/
>         /**/            col(MatStencil_c,2) = offset - 1/**/
>         /**//**/
>         /**/            value_insert(2) = -(
>         1./(cell_y(j)%pd_N+cell_y(j+1)%pd_S))*(c_zx(i,k)%fc_N)*inv_Re/**/
>         /**//**/
>         /**/            col(MatStencil_i,4) = i - 1 !south/**/
>         /**//**/
>         /**/            col(MatStencil_j,4) = j - 1 - 1/**/
>         /**//**/
>         /**/            col(MatStencil_k,4) = k - 1/**/
>         /**//**/
>         /**/            col(MatStencil_c,4) = offset - 1/**/
>         /**//**/
>         /**/            value_insert(4) = -(
>         1./(cell_y(j)%pd_S+cell_y(j-1)%pd_N))*(c_zx(i,k)%fc_N)*inv_Re/**/
>         /**//**/
>         /**/            col(MatStencil_i,6) = i - 1 !front/**/
>         /**//**/
>         /**/            col(MatStencil_j,6) = j - 1/**/
>         /**//**/
>         /**/            col(MatStencil_k,6) = k + 1 - 1/**/
>         /**//**/
>         /**/            col(MatStencil_c,6) = offset - 1/**/
>         /**//**/
>         /**/            value_insert(6) = -(
>         1./(cell_z(k)%pd_F+cell_z(k+1)%pd_B))*(c_xy(i,j)%fc_F)*inv_Re/**/
>         /**//**/
>         /**/            col(MatStencil_i,7) = i - 1 !back/**/
>         /**//**/
>         /**/            col(MatStencil_j,7) = j - 1/**/
>         /**//**/
>         /**/            col(MatStencil_k,7) = k - 1 - 1/**/
>         /**//**/
>         /**/            col(MatStencil_c,7) = offset - 1/**/
>         /**//**/
>         /**/            value_insert(7) = -(
>         1./(cell_z(k)%pd_B+cell_z(k-1)%pd_F))*(c_xy(i,j)%fc_F)*inv_Re/**/
>         /**//**/
>         /**/            col(MatStencil_i,1) = i - 1/**/
>         /**//**/
>         /**/            col(MatStencil_j,1) = j - 1/**/
>         /**//**/
>         /**/            col(MatStencil_k,1) = k - 1/**/
>         /**//**/
>         /**/            col(MatStencil_c,1) = offset - 1/**/
>         /**//**/
>         /**/            value_insert(1) = 2.*c(i,j,k)%vol/del_t -
>         (value_insert(2)+value_insert(3)+value_insert(4)+value_insert(5)+value_insert(6)+value_insert(7))/**/
>         /**//**/
>         /**/            call
>         MatSetValuesStencil(A_semi_xyz,ione,row,iseven,col,value_insert,INSERT_VALUES,ierr)/**/
>         /**//**/
>         /**/        end do/**/
>         /**//**/
>         /**/    end do/**/
>         /**/
>         /**/end do /*
>
>         Thank you
>
>         Yours sincerely,
>
>         TAY wee-beng
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150923/d18fe6c2/attachment-0001.html>

From timothee.nicolas at gmail.com  Wed Sep 23 03:45:58 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Wed, 23 Sep 2015 17:45:58 +0900
Subject: [petsc-users] Insert values into matrix using
	MatSetValuesStencil
In-Reply-To: <560264C4.70205@gmail.com>
References: <CADSzT4JLACgt_UZvAG+uyDkNaSOzs1t4QAr0T4DozWN--8Fe+w@mail.gmail.com>
	<CAGi1ndQLKN_+cUBchLD8SAqDn8QO46TQf+L69rjqqCLkXaX1UA@mail.gmail.com>
	<55DD3CC1.5070801@gmail.com>
	<CAGi1ndR78hB8M9_ybho-favYfdJnsissE_A45V1hFYZTQ0AC0w@mail.gmail.com>
	<55DD4869.2000006@gmail.com>
	<CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
	<56025F54.3040604@gmail.com>
	<CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>
	<CAGi1ndRp69mEq8VnK86Uc8HYvZ6b5zXjYdKvtqkw3niC2ew9-Q@mail.gmail.com>
	<560264C4.70205@gmail.com>
Message-ID: <CAGi1ndQ9gCf-_u9H0r=o+S8z0P4pWU2LeQLGffb_wOSt-dx8EQ@mail.gmail.com>

Yes, I had understood that, I am doing the same, but with 8 dof. This does
not change the declaration for row and col.

Can you send (i) the commands you use to create the DMDA, (ii) the commands
to create the matrix and (iii) those for the definitions of
ksta2,kend2,jsta2,jend2,size_x ?

Timoth?e

2015-09-23 17:37 GMT+09:00 TAY wee-beng <zonexo at gmail.com>:

> Hi Timoth?e,
>
> The matrix is created with 3 dof - u,v,w. So for each i,j,k, there are 3
> values. Actually I got 3 eqns from the u,v,w momentum eqns. They are not
> coupled together so I can also solve them individually. But I was told it's
> faster to group them together. The global indices is something like this:
>
> i, j, k, ijk global indices, dof
> 1 1 1 1 u
> 1 1 1 2 v
> 1 1 1 3 w
> 2 1 1 4 u
>
> ...
>
> Hope it's clearer now.
>
> Ok, I changed the stencil, it's now working.
>
> Thank you
>
> Yours sincerely,
>
> TAY wee-beng
>
> On 23/9/2015 4:22 PM, Timoth?e Nicolas wrote:
>
> Can you also tell how you created the matrix ? Just in case you created it
> with the 1 dof DMDA, it would not work if you try to input values at places
> where it is not allocated (which could explain the error message)
>
> 2015-09-23 17:18 GMT+09:00 Timoth?e Nicolas <timothee.nicolas at gmail.com>:
>
>> The first thing that strikes me is your definition of the stencils
>>
>> *MatStencil  :: row(6,1),col(6,7)*
>>
>> Why is it not defined with
>>
>> *MatStencil  :: row(4,1),col(4,7)*
>>
>> instead ?
>>
>> Where does the 6 come from ?
>>
>> Timoth?e
>>
>>
>> 2015-09-23 17:14 GMT+09:00 TAY wee-beng < <zonexo at gmail.com>
>> zonexo at gmail.com>:
>>
>>> Hi,
>>>
>>> I have successfully used MatSetValuesStencil to insert values into a
>>> Poisson eqn matrix which has 1 DOF (pressure). Now I'm trying to insert
>>> values in a momentum eqn matrix which has 3 DOF (u,v,w)
>>>
>>> However, I got the error:
>>>
>>> *[0]PETSC ERROR: --------------------- Error Message
>>> ----------------------------*
>>> *----------------------------------*
>>> *[0]PETSC ERROR: Argument out of range*
>>> *[0]PETSC ERROR: Inserting a new nonzero at (111,5) in the matrix*
>>> *[0]PETSC ERROR: See
>>> <http://www.mcs.anl.gov/petsc/documentation/faq.html>http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trou*
>>> *ble shooting.*
>>>
>>> I wonder what's wrong.  For the momentum eqn, for each DOF, at at node
>>> (dof,i,j,k), I have coupling i +/- 1, j +/- 1 and k +/- 1.
>>>
>>> The error happens at 111,5, which corresponds to i = 2, j = 2, k = 2,
>>> which is an internal node.
>>>
>>> Here's part of my code below. Hope someone can help. Thanks!
>>>
>>> *PetscScalar :: value_insert(7)*
>>>
>>> *MatStencil  :: row(6,1),col(6,7)*
>>>
>>> *ione = 1;   iseven = 7*
>>>
>>> *if (cell_type == 'u') then*
>>>
>>> *    offset = 1*
>>>
>>> *else if (cell_type == 'v') then*
>>>
>>> *    offset = 2*
>>>
>>> *else if (cell_type == 'w') then*
>>>
>>> *    offset = 3*
>>>
>>> *end if*
>>>
>>> *do k=ksta2,kend2*
>>>
>>> *    do j = jsta2,jend2*
>>>
>>> *        do i=2,size_x-1*
>>>
>>> *            row(MatStencil_i,1) = i - 1*
>>>
>>> *            row(MatStencil_j,1) = j - 1*
>>>
>>> *            row(MatStencil_k,1) = k - 1*
>>>
>>> *            row(MatStencil_c,1) = offset - 1*
>>>
>>> *            value_insert = 0.d0*
>>>
>>> *            col(MatStencil_i,3) = i + 1 - 1 !east*
>>>
>>> *            col(MatStencil_j,3) = j - 1*
>>>
>>> *            col(MatStencil_k,3) = k - 1*
>>>
>>> *            col(MatStencil_c,3) = offset - 1*
>>>
>>> *            value_insert(3) = -(
>>> 1./(cell_x(i)%pd_E+cell_x(i+1)%pd_W))*(c_yz(j,k)%fc_E)*inv_Re*
>>>
>>> *            col(MatStencil_i,5) = i - 1 - 1 !west*
>>>
>>> *            col(MatStencil_j,5) = j - 1*
>>>
>>> *            col(MatStencil_k,5) = k - 1*
>>>
>>> *            col(MatStencil_c,5) = offset - 1*
>>>
>>> *            value_insert(5) = -(
>>> 1./(cell_x(i)%pd_W+cell_x(i-1)%pd_E))*(c_yz(j,k)%fc_E)*inv_Re*
>>>
>>> *            col(MatStencil_i,2) = i - 1 !north*
>>>
>>> *            col(MatStencil_j,2) = j + 1 - 1*
>>>
>>> *            col(MatStencil_k,2) = k - 1*
>>>
>>> *            col(MatStencil_c,2) = offset - 1*
>>>
>>> *            value_insert(2) = -(
>>> 1./(cell_y(j)%pd_N+cell_y(j+1)%pd_S))*(c_zx(i,k)%fc_N)*inv_Re*
>>>
>>> *            col(MatStencil_i,4) = i - 1 !south*
>>>
>>> *            col(MatStencil_j,4) = j - 1 - 1*
>>>
>>> *            col(MatStencil_k,4) = k - 1*
>>>
>>> *            col(MatStencil_c,4) = offset - 1*
>>>
>>> *            value_insert(4) = -(
>>> 1./(cell_y(j)%pd_S+cell_y(j-1)%pd_N))*(c_zx(i,k)%fc_N)*inv_Re*
>>>
>>> *            col(MatStencil_i,6) = i - 1 !front*
>>>
>>> *            col(MatStencil_j,6) = j - 1*
>>>
>>> *            col(MatStencil_k,6) = k + 1 - 1*
>>>
>>> *            col(MatStencil_c,6) = offset - 1*
>>>
>>> *            value_insert(6) = -(
>>> 1./(cell_z(k)%pd_F+cell_z(k+1)%pd_B))*(c_xy(i,j)%fc_F)*inv_Re*
>>>
>>> *            col(MatStencil_i,7) = i - 1 !back*
>>>
>>> *            col(MatStencil_j,7) = j - 1*
>>>
>>> *            col(MatStencil_k,7) = k - 1 - 1*
>>>
>>> *            col(MatStencil_c,7) = offset - 1*
>>>
>>> *            value_insert(7) = -(
>>> 1./(cell_z(k)%pd_B+cell_z(k-1)%pd_F))*(c_xy(i,j)%fc_F)*inv_Re*
>>>
>>> *            col(MatStencil_i,1) = i - 1*
>>>
>>> *            col(MatStencil_j,1) = j - 1*
>>>
>>> *            col(MatStencil_k,1) = k - 1*
>>>
>>> *            col(MatStencil_c,1) = offset - 1*
>>>
>>> *            value_insert(1) = 2.*c(i,j,k)%vol/del_t -
>>> (value_insert(2)+value_insert(3)+value_insert(4)+value_insert(5)+value_insert(6)+value_insert(7))*
>>>
>>> *            call
>>> MatSetValuesStencil(A_semi_xyz,ione,row,iseven,col,value_insert,INSERT_VALUES,ierr)*
>>>
>>> *        end do*
>>>
>>> *    end do*
>>>
>>> *end do    *
>>>
>>> Thank you
>>>
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150923/ec871afe/attachment.html>

From zonexo at gmail.com  Wed Sep 23 04:13:09 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Wed, 23 Sep 2015 17:13:09 +0800
Subject: [petsc-users] Insert values into matrix using
	MatSetValuesStencil
In-Reply-To: <CAGi1ndQ9gCf-_u9H0r=o+S8z0P4pWU2LeQLGffb_wOSt-dx8EQ@mail.gmail.com>
References: <CADSzT4JLACgt_UZvAG+uyDkNaSOzs1t4QAr0T4DozWN--8Fe+w@mail.gmail.com>
	<CAGi1ndQLKN_+cUBchLD8SAqDn8QO46TQf+L69rjqqCLkXaX1UA@mail.gmail.com>
	<55DD3CC1.5070801@gmail.com>
	<CAGi1ndR78hB8M9_ybho-favYfdJnsissE_A45V1hFYZTQ0AC0w@mail.gmail.com>
	<55DD4869.2000006@gmail.com>
	<CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
	<56025F54.3040604@gmail.com>
	<CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>
	<CAGi1ndRp69mEq8VnK86Uc8HYvZ6b5zXjYdKvtqkw3niC2ew9-Q@mail.gmail.com>
	<560264C4.70205@gmail.com>
	<CAGi1ndQ9gCf-_u9H0r=o+S8z0P4pWU2LeQLGffb_wOSt-dx8EQ@mail.gmail.com>
Message-ID: <56026D25.2010105@gmail.com>

Hi Timoth?e,

Maybe I can send you part of it 1st. I'm trying to pinpoint why my 
matrix using MatView shows zero for a lot of the values

For i=1,j=1,k=1,

It should be :

Mat Object: 1 MPI processes
   type: seqaij
row 0: (0, 2)  (12, -2)

but now it's:

row 0: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)  
(8, 0)  (12, 0)  (13, 0)  (14, 0)  (24, 0)  (25, 0)  (26, 0) (96, 0)  
(97, 0)  (98, 0)  (192, 0)  (193, 0)  (194, 0)

I used:

/call 
DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,&//
//
//size_z,1,PETSC_DECIDE,PETSC_DECIDE,3,stencil_width,lx,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da_uvw_ast,ierr)//
//
//call DMSetMatType(da_uvw_ast,MATAIJ,ierr)//
////
//    call DMCreateMatrix(da_uvw_ast,A_semi_xyz,ierr)//
//
//    call MatSetFromOptions(A_semi_xyz,ierr)//
//
//i = 1, j = 1, k = 1//
//
//row(MatStencil_i,1) = i - 1//
////
//                    row(MatStencil_j,1) = j - 1//
////
//                    row(MatStencil_k,1) = k - 1//
////
//                    row(MatStencil_c,1) = 0//
////
//                    col(MatStencil_i,1) = i - 1//
////
//                    col(MatStencil_j,1) = j - 1//
////
//                    col(MatStencil_k,1) = k - 1//
////
//                    col(MatStencil_c,1) = 0//
////
//                    value_insert(1) = 2.//
//
//                    call 
MatSetValuesStencil(A_mat,ione,row,ione,col(:,1),value_insert(1),INSERT_VALUES,ierr)//
//
//                    col(MatStencil_i,2) = i - 1 !north//
////
//                    col(MatStencil_j,2) = j + 1 - 1//
////
//                    col(MatStencil_k,2) = k - 1//
////
//                    col(MatStencil_c,2) = 0//
////
//                    value_insert(2) = -2.//
//
//                    call 
MatSetValuesStencil(A_mat,ione,row,ione,col(:,2),value_insert(2),INSERT_VALUES,ierr)/

Thank you

Yours sincerely,

TAY wee-beng

On 23/9/2015 4:45 PM, Timoth?e Nicolas wrote:
> Yes, I had understood that, I am doing the same, but with 8 dof. This 
> does not change the declaration for row and col.
>
> Can you send (i) the commands you use to create the DMDA, (ii) the 
> commands to create the matrix and (iii) those for the definitions of 
> ksta2,kend2,jsta2,jend2,size_x ?
>
> Timoth?e
>
> 2015-09-23 17:37 GMT+09:00 TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>>:
>
>     Hi Timoth?e,
>
>     The matrix is created with 3 dof - u,v,w. So for each i,j,k, there
>     are 3 values. Actually I got 3 eqns from the u,v,w momentum eqns.
>     They are not coupled together so I can also solve them
>     individually. But I was told it's faster to group them together.
>     The global indices is something like this:
>
>     i, j, k, ijk global indices, dof
>     1 1 1 1 u
>     1 1 1 2 v
>     1 1 1 3 w
>     2 1 1 4 u
>
>     ...
>
>     Hope it's clearer now.
>
>     Ok, I changed the stencil, it's now working.
>
>     Thank you
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>     On 23/9/2015 4:22 PM, Timoth?e Nicolas wrote:
>>     Can you also tell how you created the matrix ? Just in case you
>>     created it with the 1 dof DMDA, it would not work if you try to
>>     input values at places where it is not allocated (which could
>>     explain the error message)
>>
>>     2015-09-23 17:18 GMT+09:00 Timoth?e Nicolas
>>     <timothee.nicolas at gmail.com <mailto:timothee.nicolas at gmail.com>>:
>>
>>         The first thing that strikes me is your definition of the
>>         stencils
>>
>>         */MatStencil  :: row(6,1),col(6,7)/**/
>>
>>         /*
>>         Why is it not defined with
>>
>>         */MatStencil  :: row(4,1),col(4,7)/**/
>>         /*
>>         instead ?
>>
>>         Where does the 6 come from ?
>>
>>         Timoth?e
>>
>>
>>         2015-09-23 17:14 GMT+09:00 TAY wee-beng <zonexo at gmail.com
>>         <mailto:zonexo at gmail.com>>:
>>
>>             Hi,
>>
>>             I have successfully used MatSetValuesStencil to insert
>>             values into a Poisson eqn matrix which has 1 DOF
>>             (pressure). Now I'm trying to insert values in a momentum
>>             eqn matrix which has 3 DOF (u,v,w)
>>
>>             However, I got the error:
>>
>>             /*[0]PETSC ERROR: --------------------- Error Message
>>             ----------------------------*//*
>>             *//*----------------------------------*//*
>>             *//*[0]PETSC ERROR: Argument out of range*//*
>>             *//*[0]PETSC ERROR: Inserting a new nonzero at (111,5) in
>>             the matrix*//*
>>             *//*[0]PETSC ERROR: See
>>             http://www.mcs.anl.gov/petsc/documentation/faq.html for
>>             trou*//*
>>             *//*ble shooting.*/
>>
>>             I wonder what's wrong.  For the momentum eqn, for each
>>             DOF, at at node (dof,i,j,k), I have coupling i +/- 1, j
>>             +/- 1 and k +/- 1.
>>
>>             The error happens at 111,5, which corresponds to i = 2, j
>>             = 2, k = 2, which is an internal node.
>>
>>             Here's part of my code below. Hope someone can help. Thanks!
>>             */
>>             /**/PetscScalar :: value_insert(7)/**/
>>             /**/
>>             /**/MatStencil  :: row(6,1),col(6,7)/**/
>>             /**/
>>             /**/ione = 1;   iseven = 7/**/
>>             /**/
>>             /**/if (cell_type == 'u') then/**/
>>             /**/
>>             /**/    offset = 1/**/
>>             /**//**/
>>             /**/else if (cell_type == 'v') then/**/
>>             /**/
>>             /**/    offset = 2/**/
>>             /**//**/
>>             /**/else if (cell_type == 'w') then/**/
>>             /**/
>>             /**/    offset = 3/**/
>>             /**//**/
>>             /**/end if/**/
>>             /**/
>>             /**/do k=ksta2,kend2/**/
>>             /**/
>>             /**/    do j = jsta2,jend2/**/
>>             /**/
>>             /**/        do i=2,size_x-1/**/
>>             /**//**/
>>             /**/row(MatStencil_i,1) = i - 1/**/
>>             /**//**/
>>             /**/row(MatStencil_j,1) = j - 1/**/
>>             /**//**/
>>             /**/row(MatStencil_k,1) = k - 1/**/
>>             /**//**/
>>             /**/row(MatStencil_c,1) = offset - 1/**/
>>             /**//**/
>>             /**/value_insert = 0.d0/**/
>>             /**//**/
>>             /**/col(MatStencil_i,3) = i + 1 - 1 !east/**/
>>             /**//**/
>>             /**/col(MatStencil_j,3) = j - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_k,3) = k - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_c,3) = offset - 1/**/
>>             /**//**/
>>             /**/value_insert(3) = -(
>>             1./(cell_x(i)%pd_E+cell_x(i+1)%pd_W))*(c_yz(j,k)%fc_E)*inv_Re/**/
>>             /**//**/
>>             /**/col(MatStencil_i,5) = i - 1 - 1 !west/**/
>>             /**//**/
>>             /**/col(MatStencil_j,5) = j - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_k,5) = k - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_c,5) = offset - 1/**/
>>             /**//**/
>>             /**/value_insert(5) = -(
>>             1./(cell_x(i)%pd_W+cell_x(i-1)%pd_E))*(c_yz(j,k)%fc_E)*inv_Re/**/
>>             /**//**/
>>             /**/col(MatStencil_i,2) = i - 1 !north/**/
>>             /**//**/
>>             /**/col(MatStencil_j,2) = j + 1 - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_k,2) = k - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_c,2) = offset - 1/**/
>>             /**//**/
>>             /**/value_insert(2) = -(
>>             1./(cell_y(j)%pd_N+cell_y(j+1)%pd_S))*(c_zx(i,k)%fc_N)*inv_Re/**/
>>             /**//**/
>>             /**/col(MatStencil_i,4) = i - 1 !south/**/
>>             /**//**/
>>             /**/col(MatStencil_j,4) = j - 1 - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_k,4) = k - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_c,4) = offset - 1/**/
>>             /**//**/
>>             /**/value_insert(4) = -(
>>             1./(cell_y(j)%pd_S+cell_y(j-1)%pd_N))*(c_zx(i,k)%fc_N)*inv_Re/**/
>>             /**//**/
>>             /**/col(MatStencil_i,6) = i - 1 !front/**/
>>             /**//**/
>>             /**/col(MatStencil_j,6) = j - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_k,6) = k + 1 - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_c,6) = offset - 1/**/
>>             /**//**/
>>             /**/value_insert(6) = -(
>>             1./(cell_z(k)%pd_F+cell_z(k+1)%pd_B))*(c_xy(i,j)%fc_F)*inv_Re/**/
>>             /**//**/
>>             /**/col(MatStencil_i,7) = i - 1 !back/**/
>>             /**//**/
>>             /**/col(MatStencil_j,7) = j - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_k,7) = k - 1 - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_c,7) = offset - 1/**/
>>             /**//**/
>>             /**/value_insert(7) = -(
>>             1./(cell_z(k)%pd_B+cell_z(k-1)%pd_F))*(c_xy(i,j)%fc_F)*inv_Re/**/
>>             /**//**/
>>             /**/col(MatStencil_i,1) = i - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_j,1) = j - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_k,1) = k - 1/**/
>>             /**//**/
>>             /**/col(MatStencil_c,1) = offset - 1/**/
>>             /**//**/
>>             /**/value_insert(1) = 2.*c(i,j,k)%vol/del_t -
>>             (value_insert(2)+value_insert(3)+value_insert(4)+value_insert(5)+value_insert(6)+value_insert(7))/**/
>>             /**//**/
>>             /**/            call
>>             MatSetValuesStencil(A_semi_xyz,ione,row,iseven,col,value_insert,INSERT_VALUES,ierr)/**/
>>             /**//**/
>>             /**/        end do/**/
>>             /**//**/
>>             /**/    end do/**/
>>             /**/
>>             /**/end do /*
>>
>>             Thank you
>>
>>             Yours sincerely,
>>
>>             TAY wee-beng
>>
>>
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150923/529deb3c/attachment-0001.html>

From timothee.nicolas at gmail.com  Wed Sep 23 04:25:06 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Wed, 23 Sep 2015 18:25:06 +0900
Subject: [petsc-users] Insert values into matrix using
	MatSetValuesStencil
In-Reply-To: <56026D25.2010105@gmail.com>
References: <CADSzT4JLACgt_UZvAG+uyDkNaSOzs1t4QAr0T4DozWN--8Fe+w@mail.gmail.com>
	<CAGi1ndQLKN_+cUBchLD8SAqDn8QO46TQf+L69rjqqCLkXaX1UA@mail.gmail.com>
	<55DD3CC1.5070801@gmail.com>
	<CAGi1ndR78hB8M9_ybho-favYfdJnsissE_A45V1hFYZTQ0AC0w@mail.gmail.com>
	<55DD4869.2000006@gmail.com>
	<CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
	<56025F54.3040604@gmail.com>
	<CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>
	<CAGi1ndRp69mEq8VnK86Uc8HYvZ6b5zXjYdKvtqkw3niC2ew9-Q@mail.gmail.com>
	<560264C4.70205@gmail.com>
	<CAGi1ndQ9gCf-_u9H0r=o+S8z0P4pWU2LeQLGffb_wOSt-dx8EQ@mail.gmail.com>
	<56026D25.2010105@gmail.com>
Message-ID: <CAGi1ndRxBT77xKRfPC58ewCz21_N2H16Vh2ZVfcGeq08e+ABQA@mail.gmail.com>

Hum, sorry, I don't know. I asked you to provide the definitions of start
and end values of i,j,k, because I was concerned whether you may mess up
the boundaries. Especially because you seem to treat x differently from y
and z. You have the problem also on only 1 process ?

2015-09-23 18:13 GMT+09:00 TAY wee-beng <zonexo at gmail.com>:

> Hi Timoth?e,
>
> Maybe I can send you part of it 1st. I'm trying to pinpoint why my matrix
> using MatView shows zero for a lot of the values
>
> For i=1,j=1,k=1,
>
> It should be :
>
> Mat Object: 1 MPI processes
>   type: seqaij
> row 0: (0, 2)  (12, -2)
>
> but now it's:
>
> row 0: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)  (8,
> 0)  (12, 0)  (13, 0)  (14, 0)  (24, 0)  (25, 0)  (26, 0)  (96, 0)  (97, 0)
> (98, 0)  (192, 0)  (193, 0)  (194, 0)
>
> I used:
>
> *call
> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,&*
>
>
> *size_z,1,PETSC_DECIDE,PETSC_DECIDE,3,stencil_width,lx,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da_uvw_ast,ierr)*
>
> *call DMSetMatType(da_uvw_ast,MATAIJ,ierr)*
>
> *    call DMCreateMatrix(da_uvw_ast,A_semi_xyz,ierr)*
>
> *    call MatSetFromOptions(A_semi_xyz,ierr)*
>
> *i = 1, j = 1, k = 1*
>
> *row(MatStencil_i,1) = i - 1*
>
> *                    row(MatStencil_j,1) = j - 1*
>
> *                    row(MatStencil_k,1) = k - 1*
>
> *                    row(MatStencil_c,1) = 0*
>
> *                    col(MatStencil_i,1) = i - 1*
>
> *                    col(MatStencil_j,1) = j - 1*
>
> *                    col(MatStencil_k,1) = k - 1*
>
> *                    col(MatStencil_c,1) = 0*
>
> *                    value_insert(1) = 2.*
>
> *                    call
> MatSetValuesStencil(A_mat,ione,row,ione,col(:,1),value_insert(1),INSERT_VALUES,ierr)*
>
> *                    col(MatStencil_i,2) = i - 1 !north*
>
> *                    col(MatStencil_j,2) = j + 1 - 1*
>
> *                    col(MatStencil_k,2) = k - 1*
>
> *                    col(MatStencil_c,2) = 0*
>
> *                    value_insert(2) = -2.*
>
> *                    call
> MatSetValuesStencil(A_mat,ione,row,ione,col(:,2),value_insert(2),INSERT_VALUES,ierr)*
>
> Thank you
>
> Yours sincerely,
>
> TAY wee-beng
>
> On 23/9/2015 4:45 PM, Timoth?e Nicolas wrote:
>
> Yes, I had understood that, I am doing the same, but with 8 dof. This does
> not change the declaration for row and col.
>
> Can you send (i) the commands you use to create the DMDA, (ii) the
> commands to create the matrix and (iii) those for the definitions of
> ksta2,kend2,jsta2,jend2,size_x ?
>
> Timoth?e
>
> 2015-09-23 17:37 GMT+09:00 TAY wee-beng <zonexo at gmail.com>:
>
>> Hi Timoth?e,
>>
>> The matrix is created with 3 dof - u,v,w. So for each i,j,k, there are 3
>> values. Actually I got 3 eqns from the u,v,w momentum eqns. They are not
>> coupled together so I can also solve them individually. But I was told it's
>> faster to group them together. The global indices is something like this:
>>
>> i, j, k, ijk global indices, dof
>> 1 1 1 1 u
>> 1 1 1 2 v
>> 1 1 1 3 w
>> 2 1 1 4 u
>>
>> ...
>>
>> Hope it's clearer now.
>>
>> Ok, I changed the stencil, it's now working.
>>
>> Thank you
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>> On 23/9/2015 4:22 PM, Timoth?e Nicolas wrote:
>>
>> Can you also tell how you created the matrix ? Just in case you created
>> it with the 1 dof DMDA, it would not work if you try to input values at
>> places where it is not allocated (which could explain the error message)
>>
>> 2015-09-23 17:18 GMT+09:00 Timoth?e Nicolas <
>> <timothee.nicolas at gmail.com>timothee.nicolas at gmail.com>:
>>
>>> The first thing that strikes me is your definition of the stencils
>>>
>>> *MatStencil  :: row(6,1),col(6,7)*
>>>
>>> Why is it not defined with
>>>
>>> *MatStencil  :: row(4,1),col(4,7)*
>>>
>>> instead ?
>>>
>>> Where does the 6 come from ?
>>>
>>> Timoth?e
>>>
>>>
>>> 2015-09-23 17:14 GMT+09:00 TAY wee-beng < <zonexo at gmail.com>
>>> zonexo at gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> I have successfully used MatSetValuesStencil to insert values into a
>>>> Poisson eqn matrix which has 1 DOF (pressure). Now I'm trying to insert
>>>> values in a momentum eqn matrix which has 3 DOF (u,v,w)
>>>>
>>>> However, I got the error:
>>>>
>>>> *[0]PETSC ERROR: --------------------- Error Message
>>>> ----------------------------*
>>>> *----------------------------------*
>>>> *[0]PETSC ERROR: Argument out of range*
>>>> *[0]PETSC ERROR: Inserting a new nonzero at (111,5) in the matrix*
>>>> *[0]PETSC ERROR: See
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trou*
>>>> *ble shooting.*
>>>>
>>>> I wonder what's wrong.  For the momentum eqn, for each DOF, at at node
>>>> (dof,i,j,k), I have coupling i +/- 1, j +/- 1 and k +/- 1.
>>>>
>>>> The error happens at 111,5, which corresponds to i = 2, j = 2, k = 2,
>>>> which is an internal node.
>>>>
>>>> Here's part of my code below. Hope someone can help. Thanks!
>>>>
>>>> *PetscScalar :: value_insert(7)*
>>>>
>>>> *MatStencil  :: row(6,1),col(6,7)*
>>>>
>>>> *ione = 1;   iseven = 7*
>>>>
>>>> *if (cell_type == 'u') then*
>>>>
>>>> *    offset = 1*
>>>>
>>>> *else if (cell_type == 'v') then*
>>>>
>>>> *    offset = 2*
>>>>
>>>> *else if (cell_type == 'w') then*
>>>>
>>>> *    offset = 3*
>>>>
>>>> *end if*
>>>>
>>>> *do k=ksta2,kend2*
>>>>
>>>> *    do j = jsta2,jend2*
>>>>
>>>> *        do i=2,size_x-1*
>>>>
>>>> *            row(MatStencil_i,1) = i - 1*
>>>>
>>>> *            row(MatStencil_j,1) = j - 1*
>>>>
>>>> *            row(MatStencil_k,1) = k - 1*
>>>>
>>>> *            row(MatStencil_c,1) = offset - 1*
>>>>
>>>> *            value_insert = 0.d0*
>>>>
>>>> *            col(MatStencil_i,3) = i + 1 - 1 !east*
>>>>
>>>> *            col(MatStencil_j,3) = j - 1*
>>>>
>>>> *            col(MatStencil_k,3) = k - 1*
>>>>
>>>> *            col(MatStencil_c,3) = offset - 1*
>>>>
>>>> *            value_insert(3) = -(
>>>> 1./(cell_x(i)%pd_E+cell_x(i+1)%pd_W))*(c_yz(j,k)%fc_E)*inv_Re*
>>>>
>>>> *            col(MatStencil_i,5) = i - 1 - 1 !west*
>>>>
>>>> *            col(MatStencil_j,5) = j - 1*
>>>>
>>>> *            col(MatStencil_k,5) = k - 1*
>>>>
>>>> *            col(MatStencil_c,5) = offset - 1*
>>>>
>>>> *            value_insert(5) = -(
>>>> 1./(cell_x(i)%pd_W+cell_x(i-1)%pd_E))*(c_yz(j,k)%fc_E)*inv_Re*
>>>>
>>>> *            col(MatStencil_i,2) = i - 1 !north*
>>>>
>>>> *            col(MatStencil_j,2) = j + 1 - 1*
>>>>
>>>> *            col(MatStencil_k,2) = k - 1*
>>>>
>>>> *            col(MatStencil_c,2) = offset - 1*
>>>>
>>>> *            value_insert(2) = -(
>>>> 1./(cell_y(j)%pd_N+cell_y(j+1)%pd_S))*(c_zx(i,k)%fc_N)*inv_Re*
>>>>
>>>> *            col(MatStencil_i,4) = i - 1 !south*
>>>>
>>>> *            col(MatStencil_j,4) = j - 1 - 1*
>>>>
>>>> *            col(MatStencil_k,4) = k - 1*
>>>>
>>>> *            col(MatStencil_c,4) = offset - 1*
>>>>
>>>> *            value_insert(4) = -(
>>>> 1./(cell_y(j)%pd_S+cell_y(j-1)%pd_N))*(c_zx(i,k)%fc_N)*inv_Re*
>>>>
>>>> *            col(MatStencil_i,6) = i - 1 !front*
>>>>
>>>> *            col(MatStencil_j,6) = j - 1*
>>>>
>>>> *            col(MatStencil_k,6) = k + 1 - 1*
>>>>
>>>> *            col(MatStencil_c,6) = offset - 1*
>>>>
>>>> *            value_insert(6) = -(
>>>> 1./(cell_z(k)%pd_F+cell_z(k+1)%pd_B))*(c_xy(i,j)%fc_F)*inv_Re*
>>>>
>>>> *            col(MatStencil_i,7) = i - 1 !back*
>>>>
>>>> *            col(MatStencil_j,7) = j - 1*
>>>>
>>>> *            col(MatStencil_k,7) = k - 1 - 1*
>>>>
>>>> *            col(MatStencil_c,7) = offset - 1*
>>>>
>>>> *            value_insert(7) = -(
>>>> 1./(cell_z(k)%pd_B+cell_z(k-1)%pd_F))*(c_xy(i,j)%fc_F)*inv_Re*
>>>>
>>>> *            col(MatStencil_i,1) = i - 1*
>>>>
>>>> *            col(MatStencil_j,1) = j - 1*
>>>>
>>>> *            col(MatStencil_k,1) = k - 1*
>>>>
>>>> *            col(MatStencil_c,1) = offset - 1*
>>>>
>>>> *            value_insert(1) = 2.*c(i,j,k)%vol/del_t -
>>>> (value_insert(2)+value_insert(3)+value_insert(4)+value_insert(5)+value_insert(6)+value_insert(7))*
>>>>
>>>> *            call
>>>> MatSetValuesStencil(A_semi_xyz,ione,row,iseven,col,value_insert,INSERT_VALUES,ierr)*
>>>>
>>>> *        end do*
>>>>
>>>> *    end do*
>>>>
>>>> *end do    *
>>>>
>>>> Thank you
>>>>
>>>> Yours sincerely,
>>>>
>>>> TAY wee-beng
>>>>
>>>>
>>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150923/f1b00ce5/attachment-0001.html>

From anders.jellinggaard at quantumwise.com  Wed Sep 23 09:52:28 2015
From: anders.jellinggaard at quantumwise.com (Anders Jellinggaard)
Date: Wed, 23 Sep 2015 16:52:28 +0200
Subject: [petsc-users] Trivial parallelizing in SLEPc
Message-ID: <5602BCAC.7000606@quantumwise.com>

Dear PETSc Users,

I want to find the smallest eigenpairs of a hermitian operator, A, 
implemented as a matrix-less operator.

When computing y = Ax, I need the entire vector x, and I calculate the 
entire vector y, therefore I want to avoid the ownership of x and y to 
be spread over different processes.

To what extend can this problem be parallelized in SLEPc. For instance 
can I use a block Krylov or block JD algorithm where different processes 
compute different matrix vector multiplications. If I provide the 
correct operations when constructing my MatShell, can I expect the FEAST 
algorithm to compute each contour point on a different process?

Best regards
Anders J.

From hzhang at mcs.anl.gov  Wed Sep 23 10:37:18 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Wed, 23 Sep 2015 10:37:18 -0500
Subject: [petsc-users] Trivial parallelizing in SLEPc
In-Reply-To: <5602BCAC.7000606@quantumwise.com>
References: <5602BCAC.7000606@quantumwise.com>
Message-ID: <CAGCphBs0OWV6LCYs3S9y+8PinR=sb=AtbiPwn7M-m9DLJK361Q@mail.gmail.com>

Anders:

> Dear PETSc Users,
>
> I want to find the smallest eigenpairs of a hermitian operator, A,
> implemented as a matrix-less operator.
>

Do you mean matrix-free operator?
There are slepc examples  using shell matrix:
slepc/src/eps/examples/tutorials
grep -i shell *.c
ex3.c
ex9.c
ex10.c
...
Take a look at these examples.

>
> When computing y = Ax, I need the entire vector x, and I calculate the
> entire vector y, therefore I want to avoid the ownership of x and y to be
> spread over different processes.
>

Do you mean you want  x and y owned by a single process, not distributed?
Suggest doing sequential computation first, then move to parallel execution.

>
> To what extend can this problem be parallelized in SLEPc. For instance can
> I use a block Krylov or block JD algorithm where different processes
> compute different matrix vector multiplications. If I provide the correct
> operations when constructing my MatShell, can I expect the FEAST algorithm
> to compute each contour point on a different process?
>

Slepc developer might answer this question.

Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150923/eb113f17/attachment.html>

From jroman at dsic.upv.es  Wed Sep 23 12:31:24 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 23 Sep 2015 19:31:24 +0200
Subject: [petsc-users] Trivial parallelizing in SLEPc
In-Reply-To: <CAGCphBs0OWV6LCYs3S9y+8PinR=sb=AtbiPwn7M-m9DLJK361Q@mail.gmail.com>
References: <5602BCAC.7000606@quantumwise.com>
	<CAGCphBs0OWV6LCYs3S9y+8PinR=sb=AtbiPwn7M-m9DLJK361Q@mail.gmail.com>
Message-ID: <691C0F10-15F6-40A1-9E8A-1FBC190F584A@dsic.upv.es>


> El 23/9/2015, a las 17:37, Hong <hzhang at mcs.anl.gov> escribi?:
> 
> Anders:
> Dear PETSc Users,
> 
> I want to find the smallest eigenpairs of a hermitian operator, A, implemented as a matrix-less operator.
>  
> Do you mean matrix-free operator?
> There are slepc examples  using shell matrix:
> slepc/src/eps/examples/tutorials
> grep -i shell *.c 
> ex3.c
> ex9.c
> ex10.c 
> ...
> Take a look at these examples.
> 
> When computing y = Ax, I need the entire vector x, and I calculate the entire vector y, therefore I want to avoid the ownership of x and y to be spread over different processes.
>  
> Do you mean you want  x and y owned by a single process, not distributed?
> Suggest doing sequential computation first, then move to parallel execution.
> 
> To what extend can this problem be parallelized in SLEPc. For instance can I use a block Krylov or block JD algorithm where different processes compute different matrix vector multiplications. If I provide the correct operations when constructing my MatShell, can I expect the FEAST algorithm to compute each contour point on a different process?
> 
> Slepc developer might answer this question.
> 
> Hong
> 

Parallelization in SLEPc is usually based on parallel matrix-vector and vector-vector operations, so if your matrix is sequential it cannot take advantage of parallelism. In addition, there are two exceptions that can use hierarchical parallelism, namely spectrum slicing within Krylov-Schur and the contour integral solver (CISS) - but these are useful only for specific cases. The FEAST algorithm is similar to CISS, but in the SLEPc interface to FEAST we only allow matrix-based parallelism.

Jose


From jroman at dsic.upv.es  Thu Sep 24 02:24:17 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 24 Sep 2015 09:24:17 +0200
Subject: [petsc-users] Trivial parallelizing in SLEPc
In-Reply-To: <56039C30.7000707@quantumwise.com>
References: <5602BCAC.7000606@quantumwise.com>
	<CAGCphBs0OWV6LCYs3S9y+8PinR=sb=AtbiPwn7M-m9DLJK361Q@mail.gmail.com>
	<691C0F10-15F6-40A1-9E8A-1FBC190F584A@dsic.upv.es>
	<56039C30.7000707@quantumwise.com>
Message-ID: <479689F2-0C07-4BD2-A62A-AD17811DDF37@dsic.upv.es>


> El 24/9/2015, a las 8:46, Anders Jellinggaard <anders.jellinggaard at quantumwise.com> escribi?:
> 
> Thank you Jose and Hong
> 
> Spectrum slicing is described in Sec. 3.4.5 of the user manual, but is there anywhere I can read about hierarchical parallelism in the CISS solver in SLEPc?
> 
> Anders
> 

[Please reply to the list.]


At some point we will write a technical report describing the method and the user interface, but it is not available yet. You will have to look at the source code. Basically, you can set the number of partitions (or subcommunicators), and if it is equal to the number of MPI processes then the computation associated with any integration point will be done sequentially.
http://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSCISSSetSizes.html

Jose

From gideon.simpson at gmail.com  Thu Sep 24 09:50:17 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Thu, 24 Sep 2015 10:50:17 -0400
Subject: [petsc-users] inconsistent SNES/KSP behavior
Message-ID: <8E4606B1-637A-4FCF-A98E-E6E1559A3273@gmail.com>

So I may be imagining this, but I like to think that the following happened.  I submitted the same job script to my cluster, twice, and I got different results.    This is a SNES problem where I am using the MUMPS linear solver.  Here is what I mean:

Trial 1:

Generating initial guess with a0 = 1.95, b0 = 2.09, sigma = 2, xmax = 100, nx = 2560000

2560001
10240006
Writing to disk as guess.bin

 Solving sigma = 2
  0 SNES Function norm 1.691829175711e-01 
    0 KSP preconditioned resid norm 6.098266966934e-04 true resid norm 1.691829175711e-01 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.698748932236e-12 true resid norm 5.163872517578e-10 ||r(i)||/||b|| 3.052242266367e-09
  1 SNES Function norm 1.395434035402e-04 
    0 KSP preconditioned resid norm 7.922333994913e-07 true resid norm 1.395434035402e-04 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.839375186797e-14 true resid norm 3.034980160221e-13 ||r(i)||/||b|| 2.174936316030e-09
  2 SNES Function norm 1.394208676802e-04 
 sigma = 2, a = 1.94758, b = 2.09002, reason =4

Trial 2:

Generating initial guess with a0 = 1.95, b0 = 2.09, sigma = 2, xmax = 100, nx = 2560000

2560001
10240006
Writing to disk as guess.bin

 Solving sigma = 2
  0 SNES Function norm 1.691829175711e-01 
    0 KSP preconditioned resid norm 6.098266972456e-04 true resid norm 1.691829175711e-01 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.497022256767e-11 true resid norm 1.953323112042e-09 ||r(i)||/||b|| 1.154562848357e-08
  1 SNES Function norm 1.396785292629e-04 
    0 KSP preconditioned resid norm 1.430883443301e-05 true resid norm 1.396785292629e-04 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.226190977119e-14 true resid norm 3.918710626146e-12 ||r(i)||/||b|| 2.805521111100e-08
  2 SNES Function norm 1.396634713981e-04 
    0 KSP preconditioned resid norm 1.693020976611e-05 true resid norm 1.396634713981e-04 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 6.991045787498e-14 true resid norm 4.339991163168e-12 ||r(i)||/||b|| 3.107463332911e-08
  3 SNES Function norm 1.395099855381e-04 
    0 KSP preconditioned resid norm 8.023057268371e-06 true resid norm 1.395099855381e-04 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.476126449145e-14 true resid norm 2.800497344039e-12 ||r(i)||/||b|| 2.007381287610e-08
  4 SNES Function norm 1.394785126062e-04 
 sigma = 2, a = 1.94758, b = 2.09002, reason =4

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/d24648a6/attachment.html>

From knepley at gmail.com  Thu Sep 24 10:01:24 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 24 Sep 2015 10:01:24 -0500
Subject: [petsc-users] inconsistent SNES/KSP behavior
In-Reply-To: <8E4606B1-637A-4FCF-A98E-E6E1559A3273@gmail.com>
References: <8E4606B1-637A-4FCF-A98E-E6E1559A3273@gmail.com>
Message-ID: <CAMYG4GnOb71srmOGdNAeozfU14XpUrVB0OEOuPiDrER8BvAN-w@mail.gmail.com>

On Thu, Sep 24, 2015 at 9:50 AM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> So I may be imagining this, but I like to think that the following
> happened.  I submitted the same job script to my cluster, twice, and I got
> different results.    This is a SNES problem where I am using the MUMPS
> linear solver.  Here is what I mean:
>

It looks like MUMPS uses randomization, since the initial true residuals
are the same, but the preconditioned is different.

  Matt


> *Trial 1:*
>
> Generating initial guess with a0 = 1.95, b0 = 2.09, sigma = 2, xmax = 100,
> nx = 2560000
>
> 2560001
> 10240006
> Writing to disk as guess.bin
>
>  Solving sigma = 2
>   0 SNES Function norm 1.691829175711e-01
>     0 KSP preconditioned resid norm 6.098266966934e-04 true resid norm
> 1.691829175711e-01 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.698748932236e-12 true resid norm
> 5.163872517578e-10 ||r(i)||/||b|| 3.052242266367e-09
>   1 SNES Function norm 1.395434035402e-04
>     0 KSP preconditioned resid norm 7.922333994913e-07 true resid norm
> 1.395434035402e-04 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.839375186797e-14 true resid norm
> 3.034980160221e-13 ||r(i)||/||b|| 2.174936316030e-09
>   2 SNES Function norm 1.394208676802e-04
>  sigma = 2, a = 1.94758, b = 2.09002, reason =4
>
> *Trial 2:*
>
> Generating initial guess with a0 = 1.95, b0 = 2.09, sigma = 2, xmax = 100,
> nx = 2560000
>
> 2560001
> 10240006
> Writing to disk as guess.bin
>
>  Solving sigma = 2
>   0 SNES Function norm 1.691829175711e-01
>     0 KSP preconditioned resid norm 6.098266972456e-04 true resid norm
> 1.691829175711e-01 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.497022256767e-11 true resid norm
> 1.953323112042e-09 ||r(i)||/||b|| 1.154562848357e-08
>   1 SNES Function norm 1.396785292629e-04
>     0 KSP preconditioned resid norm 1.430883443301e-05 true resid norm
> 1.396785292629e-04 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.226190977119e-14 true resid norm
> 3.918710626146e-12 ||r(i)||/||b|| 2.805521111100e-08
>   2 SNES Function norm 1.396634713981e-04
>     0 KSP preconditioned resid norm 1.693020976611e-05 true resid norm
> 1.396634713981e-04 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.991045787498e-14 true resid norm
> 4.339991163168e-12 ||r(i)||/||b|| 3.107463332911e-08
>   3 SNES Function norm 1.395099855381e-04
>     0 KSP preconditioned resid norm 8.023057268371e-06 true resid norm
> 1.395099855381e-04 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.476126449145e-14 true resid norm
> 2.800497344039e-12 ||r(i)||/||b|| 2.007381287610e-08
>   4 SNES Function norm 1.394785126062e-04
>  sigma = 2, a = 1.94758, b = 2.09002, reason =4
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/75de7ec6/attachment.html>

From soumyamechanics at gmail.com  Thu Sep 24 11:01:48 2015
From: soumyamechanics at gmail.com (Soumya Mukherjee)
Date: Thu, 24 Sep 2015 12:01:48 -0400
Subject: [petsc-users] number of converged eigenpairs
Message-ID: <CAPuCtQEPWUknwAO5Pwefxws4jCmuPENU7CC4g3FeXvvVYj+TyQ@mail.gmail.com>

 Hi.

The following code generates nconv numbers of eigenpairs. This number, for
my case is so few. And I am afraid, there are more important eigenmodes. Is
it possible to generate all (or more) eigenpairs?

Also, I was wondering, which eigenvalues/ eigenvectors does it calculate in
this code, whose number is given by nconv?

    MatCreateSeqAIJ( MPI_COMM_SELF, Natom * Norbit, Natom * Norbit,
Nneighbor * Norbit, PETSC_NULL, &H );
    MatSetFromOptions( H );

    MatCreateSeqAIJ( MPI_COMM_SELF, Natom * Norbit, Natom * Norbit,
Nneighbor * Norbit, PETSC_NULL, &S );
    MatSetFromOptions( S );

Hamiltonian ( );
// MatView(H,PETSC_VIEWER_STDOUT_WORLD);
// VecView(H0,PETSC_VIEWER_STDOUT_WORLD);

//-------------------------------------------------------------------------------------------------------
    /*Setup the Ax=cBx generalized eigenproblem*/

//-------------------------------------------------------------------------------------------------------
    EPSCreate( PETSC_COMM_WORLD, &eps );
    EPSSetOperators( eps, H, S);
    MatGetVecs(H,NULL,&xr);
    MatGetVecs(H,NULL,&xi);

    EPSGetST( eps, &st);
    STSetType(st,STCAYLEY);
    STCayleySetAntishift(st,1);

    EPSSetDimensions( eps, nev, ncv, PETSC_DECIDE);
    EPSSetTarget( eps,5.0);
    EPSSetFromOptions( eps );
    EPSSolve( eps );
    EPSPrintSolution(eps,PETSC_NULL);
    EPSGetConverged( eps, &nconv );

    eigenval = new complex<double>[ nconv ];
    eigenvec = new complex<double>*[ nconv ];
    for (int i = 0; i < nconv; i ++ ) {
        eigenvec[ i ] = new complex<double>[ Norbit * Natom ];
    }

    for ( jjj = 0; jjj < nconv; jjj ++ ) {
        /* Get converged eigenpairs: j-th eigenvalue is stored in kr (real
part) and ki (imaginary part) */
        EPSGetEigenpair(eps, jjj, &kr, &ki, xr, xi);
        eigenval[ jjj ] = complex
<double>(PetscRealPart(kr),PetscImaginaryPart(kr));

        VecGetArray(xr, &Xr);
        for ( int i = 0; i < Norbit * Natom; i ++ ) {
            eigenvec[ jjj ][ i ] = complex
<double>(PetscRealPart(Xr[i]),PetscImaginaryPart(Xr[i]));
        }
        VecRestoreArray(xr, &Xr);
    }
//--------------------------
/*finalizing*/
//--------------------------
outputs( true );
    VecDestroy(&xr);
    VecDestroy(&xi);
    EPSDestroy(&eps);
    MatDestroy(&H);
    MatDestroy(&S);
    SlepcFinalize();
    PetscFinalize();
return 0;
}

Regards,
Soumya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/628be04c/attachment-0001.html>

From jroman at dsic.upv.es  Thu Sep 24 15:50:25 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 24 Sep 2015 22:50:25 +0200
Subject: [petsc-users] number of converged eigenpairs
In-Reply-To: <CAPuCtQEPWUknwAO5Pwefxws4jCmuPENU7CC4g3FeXvvVYj+TyQ@mail.gmail.com>
References: <CAPuCtQEPWUknwAO5Pwefxws4jCmuPENU7CC4g3FeXvvVYj+TyQ@mail.gmail.com>
Message-ID: <AB8E38BD-0EAD-48BF-808D-9817CC94F735@dsic.upv.es>

By default only 1 eigenpair is computed. Use EPSSetDimensions() to specify how many eigenvalues you want. This is explained in the manual in basic examples.
http://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSSetDimensions.html

SLEPc is not intended for computing all eigenvalues. If you need all eigenvalues you should use another software.

In your code, the computed eigenvalues are those closest to the target.

Jose

> El 24/9/2015, a las 18:01, Soumya Mukherjee <soumyamechanics at gmail.com> escribi?:
> 
>  Hi.
> 
> The following code generates nconv numbers of eigenpairs. This number, for my case is so few. And I am afraid, there are more important eigenmodes. Is it possible to generate all (or more) eigenpairs? 
> 
> Also, I was wondering, which eigenvalues/ eigenvectors does it calculate in this code, whose number is given by nconv?
> 
>     MatCreateSeqAIJ( MPI_COMM_SELF, Natom * Norbit, Natom * Norbit, Nneighbor * Norbit, PETSC_NULL, &H );
>     MatSetFromOptions( H );
>     
>     MatCreateSeqAIJ( MPI_COMM_SELF, Natom * Norbit, Natom * Norbit, Nneighbor * Norbit, PETSC_NULL, &S );
>     MatSetFromOptions( S );
>     
> 	Hamiltonian ( );
> //	MatView(H,PETSC_VIEWER_STDOUT_WORLD);
> //	VecView(H0,PETSC_VIEWER_STDOUT_WORLD);
>     //-------------------------------------------------------------------------------------------------------
>     /*Setup the Ax=cBx generalized eigenproblem*/
>     //-------------------------------------------------------------------------------------------------------
>     EPSCreate( PETSC_COMM_WORLD, &eps );
>     EPSSetOperators( eps, H, S);
>     MatGetVecs(H,NULL,&xr);
>     MatGetVecs(H,NULL,&xi);
> 
>     EPSGetST( eps, &st);
>     STSetType(st,STCAYLEY);
>     STCayleySetAntishift(st,1);
>     
>     EPSSetDimensions( eps, nev, ncv, PETSC_DECIDE);
>     EPSSetTarget( eps,5.0);
>     EPSSetFromOptions( eps );
>     EPSSolve( eps );
>     EPSPrintSolution(eps,PETSC_NULL);
>     EPSGetConverged( eps, &nconv );
>     
>     eigenval = new complex<double>[ nconv ];
>     eigenvec = new complex<double>*[ nconv ];
>     for (int i = 0; i < nconv; i ++ ) {
>         eigenvec[ i ] = new complex<double>[ Norbit * Natom ];
>     }
>     
>     for ( jjj = 0; jjj < nconv; jjj ++ ) {
>         /*		Get converged eigenpairs: j-th eigenvalue is stored in kr (real part) and ki (imaginary part)	*/
>         EPSGetEigenpair(eps, jjj, &kr, &ki, xr, xi);
>         eigenval[ jjj ] = complex <double>(PetscRealPart(kr),PetscImaginaryPart(kr));
>         
>         VecGetArray(xr, &Xr);
>         for ( int i = 0; i < Norbit * Natom; i ++ ) {
>             eigenvec[ jjj ][ i ] = complex <double>(PetscRealPart(Xr[i]),PetscImaginaryPart(Xr[i]));
>         }
>         VecRestoreArray(xr, &Xr);
>     }
> 	
> 	//--------------------------
> 	/*finalizing*/
> 	//--------------------------
> 	
> 	outputs( true );
>     VecDestroy(&xr);
>     VecDestroy(&xi);
>     EPSDestroy(&eps);
>     MatDestroy(&H);
>     MatDestroy(&S);
>     SlepcFinalize();
>     PetscFinalize();
> 	
> 	return 0;
> }
> 
> Regards,
> Soumya


From aph at email.arizona.edu  Thu Sep 24 16:13:41 2015
From: aph at email.arizona.edu (Anthony Paul Haas)
Date: Thu, 24 Sep 2015 14:13:41 -0700
Subject: [petsc-users] SEGV
Message-ID: <CAEyxMWVf5wg9nN4hU1FWzyzpAkNROr+h9R6v8RHSaB4zQnYv6w@mail.gmail.com>

Hi,

I am running a Petsc/Slepc program on a supercomputer. The program works
fine for grid sizes 101x101, 201x201 but for 301x301, I get a SEGV (see at
the bottom of test.out). I ran the program for smaller grid sizes on my
local machine through Gnu debugger and Valgrind and everything seems fine
there.

Thanks in advance,

Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/ae4bfc7a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.out
Type: application/octet-stream
Size: 32296 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/ae4bfc7a/attachment-0001.obj>

From bsmith at mcs.anl.gov  Thu Sep 24 16:35:39 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 24 Sep 2015 16:35:39 -0500
Subject: [petsc-users] SEGV
In-Reply-To: <CAEyxMWVf5wg9nN4hU1FWzyzpAkNROr+h9R6v8RHSaB4zQnYv6w@mail.gmail.com>
References: <CAEyxMWVf5wg9nN4hU1FWzyzpAkNROr+h9R6v8RHSaB4zQnYv6w@mail.gmail.com>
Message-ID: <ED41E6DC-A396-499D-B4E5-0670E690CCA4@mcs.anl.gov>


  Sherry's last release of SuperLU_DIST had a lot of bugs that she later fixed without changing the version number. Please get the tarball directly from
http://crd-legacy.lbl.gov/~xiaoye/SuperLU/#superlu_dist  and rebuild PETSc (delete the entire PETSc directory and start from scratch to make you only get new stuff) and let us know if the problem goes away.

   Barry

> On Sep 24, 2015, at 4:13 PM, Anthony Paul Haas <aph at email.arizona.edu> wrote:
> 
> Hi,
> 
> I am running a Petsc/Slepc program on a supercomputer. The program works fine for grid sizes 101x101, 201x201 but for 301x301, I get a SEGV (see at the bottom of test.out). I ran the program for smaller grid sizes on my local machine through Gnu debugger and Valgrind and everything seems fine there.
> 
> Thanks in advance,
> 
> Anthony
> 
> 
> <test.out>


From gideon.simpson at gmail.com  Thu Sep 24 17:23:40 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Thu, 24 Sep 2015 18:23:40 -0400
Subject: [petsc-users] SNES stuck with small residual
Message-ID: <DDE3BB2E-8CED-47B0-9BE8-9040666B0832@gmail.com>

Just wondering if anyone has any suggestions for this scenario:

  0 SNES Function norm 5.979113140046e+02 
    0 KSP preconditioned resid norm 1.609660749629e+03 true resid norm 5.979113140046e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.083640787055e-06 true resid norm 4.070012556358e-04 ||r(i)||/||b|| 6.807050579289e-07
  1 SNES Function norm 5.106334303399e+02 
    0 KSP preconditioned resid norm 1.304622870569e+03 true resid norm 5.106334303399e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.542022975926e-06 true resid norm 3.284828506619e-04 ||r(i)||/||b|| 6.432850478341e-07
  2 SNES Function norm 4.522093337153e+02 
    0 KSP preconditioned resid norm 1.017788935768e+03 true resid norm 4.522093337153e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.338757544659e-06 true resid norm 2.556045980853e-04 ||r(i)||/||b|| 5.652351223829e-07
  3 SNES Function norm 4.042911112972e+02 
    0 KSP preconditioned resid norm 7.996536035697e+02 true resid norm 4.042911112972e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 8.612178921208e-07 true resid norm 2.026756625099e-04 ||r(i)||/||b|| 5.013112008810e-07
  4 SNES Function norm 3.618952927134e+02 
    0 KSP preconditioned resid norm 6.392806337991e+02 true resid norm 3.618952927134e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.099112497936e-07 true resid norm 1.605851784448e-04 ||r(i)||/||b|| 4.437338138353e-07
  5 SNES Function norm 3.239307814846e+02 
    0 KSP preconditioned resid norm 5.205889419531e+02 true resid norm 3.239307814846e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.364948858968e-07 true resid norm 1.326652268287e-04 ||r(i)||/||b|| 4.095480714142e-07
  6 SNES Function norm 2.899625386925e+02 
    0 KSP preconditioned resid norm 4.305631057215e+02 true resid norm 2.899625386925e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.472775427751e-07 true resid norm 1.094956512355e-04 ||r(i)||/||b|| 3.776199909453e-07
  7 SNES Function norm 2.596235386610e+02 
    0 KSP preconditioned resid norm 3.606693697722e+02 true resid norm 2.596235386610e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.804499181789e-07 true resid norm 9.162646660567e-05 ||r(i)||/||b|| 3.529204904849e-07
  8 SNES Function norm 2.178613867932e+02 
    0 KSP preconditioned resid norm 2.791607776778e+02 true resid norm 2.178613867932e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.269331257925e-07 true resid norm 7.151934347788e-05 ||r(i)||/||b|| 3.282791160499e-07
  9 SNES Function norm 1.541279494063e+02 
    0 KSP preconditioned resid norm 1.781925534106e+02 true resid norm 1.541279494063e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.095285568305e-07 true resid norm 4.530373014757e-05 ||r(i)||/||b|| 2.939358521416e-07
 10 SNES Function norm 1.361917300390e+02 
    0 KSP preconditioned resid norm 4.801281559143e+01 true resid norm 1.361917300390e+02 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.158004981151e-07 true resid norm 1.216364616255e-05 ||r(i)||/||b|| 8.931266354477e-08
 11 SNES Function norm 6.951420638296e+00 
    0 KSP preconditioned resid norm 2.808725633599e+00 true resid norm 6.951420638296e+00 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.361764665651e-09 true resid norm 7.088079744061e-07 ||r(i)||/||b|| 1.019659162188e-07
 12 SNES Function norm 3.249486326694e-03 
    0 KSP preconditioned resid norm 1.388995922430e-03 true resid norm 3.249486326694e-03 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 1.757348050151e-12 true resid norm 3.473453149942e-10 ||r(i)||/||b|| 1.068923762322e-07
 13 SNES Function norm 4.590368792536e-05 
    0 KSP preconditioned resid norm 4.360884570719e-06 true resid norm 4.590368792536e-05 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.926789585311e-15 true resid norm 1.102338507065e-12 ||r(i)||/||b|| 2.401416001385e-08
 14 SNES Function norm 4.422797479565e-05 
    0 KSP preconditioned resid norm 5.831925614907e-06 true resid norm 4.422797479565e-05 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.584144701532e-15 true resid norm 1.478006791605e-12 ||r(i)||/||b|| 3.341791701821e-08
 15 SNES Function norm 4.371345326958e-05 
    0 KSP preconditioned resid norm 4.897666437748e-06 true resid norm 4.371345326958e-05 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 4.080247182926e-15 true resid norm 1.236587295819e-12 ||r(i)||/||b|| 2.828848336902e-08
 16 SNES Function norm 4.348609553058e-05 
    0 KSP preconditioned resid norm 3.738450099362e-06 true resid norm 4.348609553058e-05 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 3.655406222015e-15 true resid norm 9.403440338690e-13 ||r(i)||/||b|| 2.162401619174e-08
 17 SNES Function norm 4.324907311342e-05 
    0 KSP preconditioned resid norm 5.296206632183e-06 true resid norm 4.324907311342e-05 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.018283198612e-15 true resid norm 1.334862856428e-12 ||r(i)||/||b|| 3.086454252850e-08
 18 SNES Function norm 4.324904992696e-05 
    0 KSP preconditioned resid norm 5.289306428405e-06 true resid norm 4.324904992696e-05 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.691985954854e-15 true resid norm 1.335680822478e-12 ||r(i)||/||b|| 3.088347200075e-08
 19 SNES Function norm 4.324904992565e-05 
    0 KSP preconditioned resid norm 5.289311834250e-06 true resid norm 4.324904992565e-05 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 5.455286406170e-15 true resid norm 1.337387995906e-12 ||r(i)||/||b|| 3.092294508677e-08
 

At this point, it terminates with a diverged line search.  


-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/c3c5bc1f/attachment.html>

From knepley at gmail.com  Thu Sep 24 17:28:03 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 24 Sep 2015 17:28:03 -0500
Subject: [petsc-users] SNES stuck with small residual
In-Reply-To: <DDE3BB2E-8CED-47B0-9BE8-9040666B0832@gmail.com>
References: <DDE3BB2E-8CED-47B0-9BE8-9040666B0832@gmail.com>
Message-ID: <CAMYG4GmQbUfUOCva_AfmWjTYB+LMgjeujzPX1g6XCNuewUw+8w@mail.gmail.com>

On Thu, Sep 24, 2015 at 5:23 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Just wondering if anyone has any suggestions for this scenario:
>

I have seen this happen a bunch. Your Newton directions are crap. That
means

  a) you have a crap Jacobian

  b) your Jacobian direction is crap (unlikely here)

  c) your Jacobian is very ill-conditioned

Something to try in order to see what is happening is to start doing Picard
when Newton stalls. If it
converges, then you have something like c) I think. We can try to mitigate
this by NPC. Something
as simple as Newton + NCG or Newton + NRichardson can work.

  Matt


>   0 SNES Function norm 5.979113140046e+02
>     0 KSP preconditioned resid norm 1.609660749629e+03 true resid norm
> 5.979113140046e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.083640787055e-06 true resid norm
> 4.070012556358e-04 ||r(i)||/||b|| 6.807050579289e-07
>   1 SNES Function norm 5.106334303399e+02
>     0 KSP preconditioned resid norm 1.304622870569e+03 true resid norm
> 5.106334303399e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.542022975926e-06 true resid norm
> 3.284828506619e-04 ||r(i)||/||b|| 6.432850478341e-07
>   2 SNES Function norm 4.522093337153e+02
>     0 KSP preconditioned resid norm 1.017788935768e+03 true resid norm
> 4.522093337153e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.338757544659e-06 true resid norm
> 2.556045980853e-04 ||r(i)||/||b|| 5.652351223829e-07
>   3 SNES Function norm 4.042911112972e+02
>     0 KSP preconditioned resid norm 7.996536035697e+02 true resid norm
> 4.042911112972e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 8.612178921208e-07 true resid norm
> 2.026756625099e-04 ||r(i)||/||b|| 5.013112008810e-07
>   4 SNES Function norm 3.618952927134e+02
>     0 KSP preconditioned resid norm 6.392806337991e+02 true resid norm
> 3.618952927134e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.099112497936e-07 true resid norm
> 1.605851784448e-04 ||r(i)||/||b|| 4.437338138353e-07
>   5 SNES Function norm 3.239307814846e+02
>     0 KSP preconditioned resid norm 5.205889419531e+02 true resid norm
> 3.239307814846e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.364948858968e-07 true resid norm
> 1.326652268287e-04 ||r(i)||/||b|| 4.095480714142e-07
>   6 SNES Function norm 2.899625386925e+02
>     0 KSP preconditioned resid norm 4.305631057215e+02 true resid norm
> 2.899625386925e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.472775427751e-07 true resid norm
> 1.094956512355e-04 ||r(i)||/||b|| 3.776199909453e-07
>   7 SNES Function norm 2.596235386610e+02
>     0 KSP preconditioned resid norm 3.606693697722e+02 true resid norm
> 2.596235386610e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.804499181789e-07 true resid norm
> 9.162646660567e-05 ||r(i)||/||b|| 3.529204904849e-07
>   8 SNES Function norm 2.178613867932e+02
>     0 KSP preconditioned resid norm 2.791607776778e+02 true resid norm
> 2.178613867932e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.269331257925e-07 true resid norm
> 7.151934347788e-05 ||r(i)||/||b|| 3.282791160499e-07
>   9 SNES Function norm 1.541279494063e+02
>     0 KSP preconditioned resid norm 1.781925534106e+02 true resid norm
> 1.541279494063e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.095285568305e-07 true resid norm
> 4.530373014757e-05 ||r(i)||/||b|| 2.939358521416e-07
>  10 SNES Function norm 1.361917300390e+02
>     0 KSP preconditioned resid norm 4.801281559143e+01 true resid norm
> 1.361917300390e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.158004981151e-07 true resid norm
> 1.216364616255e-05 ||r(i)||/||b|| 8.931266354477e-08
>  11 SNES Function norm 6.951420638296e+00
>     0 KSP preconditioned resid norm 2.808725633599e+00 true resid norm
> 6.951420638296e+00 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.361764665651e-09 true resid norm
> 7.088079744061e-07 ||r(i)||/||b|| 1.019659162188e-07
>  12 SNES Function norm 3.249486326694e-03
>     0 KSP preconditioned resid norm 1.388995922430e-03 true resid norm
> 3.249486326694e-03 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.757348050151e-12 true resid norm
> 3.473453149942e-10 ||r(i)||/||b|| 1.068923762322e-07
>  13 SNES Function norm 4.590368792536e-05
>     0 KSP preconditioned resid norm 4.360884570719e-06 true resid norm
> 4.590368792536e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.926789585311e-15 true resid norm
> 1.102338507065e-12 ||r(i)||/||b|| 2.401416001385e-08
>  14 SNES Function norm 4.422797479565e-05
>     0 KSP preconditioned resid norm 5.831925614907e-06 true resid norm
> 4.422797479565e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.584144701532e-15 true resid norm
> 1.478006791605e-12 ||r(i)||/||b|| 3.341791701821e-08
>  15 SNES Function norm 4.371345326958e-05
>     0 KSP preconditioned resid norm 4.897666437748e-06 true resid norm
> 4.371345326958e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.080247182926e-15 true resid norm
> 1.236587295819e-12 ||r(i)||/||b|| 2.828848336902e-08
>  16 SNES Function norm 4.348609553058e-05
>     0 KSP preconditioned resid norm 3.738450099362e-06 true resid norm
> 4.348609553058e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.655406222015e-15 true resid norm
> 9.403440338690e-13 ||r(i)||/||b|| 2.162401619174e-08
>  17 SNES Function norm 4.324907311342e-05
>     0 KSP preconditioned resid norm 5.296206632183e-06 true resid norm
> 4.324907311342e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.018283198612e-15 true resid norm
> 1.334862856428e-12 ||r(i)||/||b|| 3.086454252850e-08
>  18 SNES Function norm 4.324904992696e-05
>     0 KSP preconditioned resid norm 5.289306428405e-06 true resid norm
> 4.324904992696e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.691985954854e-15 true resid norm
> 1.335680822478e-12 ||r(i)||/||b|| 3.088347200075e-08
>  19 SNES Function norm 4.324904992565e-05
>     0 KSP preconditioned resid norm 5.289311834250e-06 true resid norm
> 4.324904992565e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.455286406170e-15 true resid norm
> 1.337387995906e-12 ||r(i)||/||b|| 3.092294508677e-08
>
>
>
> At this point, it terminates with a diverged line search.
>
>
> -gideon
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/54c269e0/attachment-0001.html>

From bsmith at mcs.anl.gov  Thu Sep 24 17:44:48 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 24 Sep 2015 17:44:48 -0500
Subject: [petsc-users] SNES stuck with small residual
In-Reply-To: <CAMYG4GmQbUfUOCva_AfmWjTYB+LMgjeujzPX1g6XCNuewUw+8w@mail.gmail.com>
References: <DDE3BB2E-8CED-47B0-9BE8-9040666B0832@gmail.com>
	<CAMYG4GmQbUfUOCva_AfmWjTYB+LMgjeujzPX1g6XCNuewUw+8w@mail.gmail.com>
Message-ID: <A5A1A71A-9EF3-4E34-BC3B-9C1C789F30A0@mcs.anl.gov>


  I completely disagree with Matt, looking at the numbers one sees very typical Newton behavior

a little slow at the beginning because it is not in the quadratic basin

0 SNES Function norm 5.979113140046e+02 
  1 SNES Function norm 5.106334303399e+02 
  2 SNES Function norm 4.522093337153e+02 
  3 SNES Function norm 4.042911112972e+02 
  4 SNES Function norm 3.618952927134e+02 
  5 SNES Function norm 3.239307814846e+02 
  6 SNES Function norm 2.899625386925e+02 
  7 SNES Function norm 2.596235386610e+02 
  8 SNES Function norm 2.178613867932e+02 
  9 SNES Function norm 1.541279494063e+02 
 10 SNES Function norm 1.361917300390e+02 

 suddenly fast convergence as it gets in the quadratic basin and behaves like Newton should behave

 11 SNES Function norm 6.951420638296e+00 
 12 SNES Function norm 3.249486326694e-03 
 13 SNES Function norm 4.590368792536e-05 

no more convergence as it "runs out of digits" 

 14 SNES Function norm 4.422797479565e-05 
 15 SNES Function norm 4.371345326958e-05 
 16 SNES Function norm 4.348609553058e-05 
 17 SNES Function norm 4.324907311342e-05 
 18 SNES Function norm 4.324904992696e-05 
 19 SNES Function norm 4.324904992565e-05 

The only difference between this run and most other runs of Newton (when it works in the standard way) is that it "runs out of digits" early (after only reducing the residual norm by 1e-7 when for super nice problems it can do like 1e-12). 

I don't think your Jacobian is crap, I don't think the Jacobian direction is crap and the Jacobian is not not terribly ill-conditioned since the direct solver reduces the residual norm by 1e-8.

My recommendation when Newton "runs out of digits" is to try using quad precision. With PETSc you just need a recent version of the GNU compilers and the configure options --with-precision=__float128 --download-f2cblaslapack   plus you need to make sure your code always uses PetscReal and PetscScalar instead of "double" or whatever and whenever you call elementary functions like sin() you call PetscSinReal() or PetscSinScalar() etc so that it uses the quad precision version (if you miss one of these you will never see the quad precision). 

  Barry

What does it mean for Newton to "run out of digits"; this is not a technical term but I believe it is related to how accurate your function evaluations can be (I think some people call this noise) when expressed using floating point numbers. Perhaps it is something like "ill-conditioned functions". If the function evaluations are only valid for 8 digits then it is not reasonable to expect Newton (or any floating point algorithm) to squeeze the norm of the function values to 1e.-12.





> On Sep 24, 2015, at 5:28 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Thu, Sep 24, 2015 at 5:23 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> Just wondering if anyone has any suggestions for this scenario:
> 
> I have seen this happen a bunch. Your Newton directions are crap. That means 
> 
>   a) you have a crap Jacobian
> 
>   b) your Jacobian direction is crap (unlikely here)
> 
>   c) your Jacobian is very ill-conditioned
> 
> Something to try in order to see what is happening is to start doing Picard when Newton stalls. If it
> converges, then you have something like c) I think. We can try to mitigate this by NPC. Something
> as simple as Newton + NCG or Newton + NRichardson can work.
> 
>   Matt
>  
>   0 SNES Function norm 5.979113140046e+02 
>     0 KSP preconditioned resid norm 1.609660749629e+03 true resid norm 5.979113140046e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.083640787055e-06 true resid norm 4.070012556358e-04 ||r(i)||/||b|| 6.807050579289e-07
>   1 SNES Function norm 5.106334303399e+02 
>     0 KSP preconditioned resid norm 1.304622870569e+03 true resid norm 5.106334303399e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.542022975926e-06 true resid norm 3.284828506619e-04 ||r(i)||/||b|| 6.432850478341e-07
>   2 SNES Function norm 4.522093337153e+02 
>     0 KSP preconditioned resid norm 1.017788935768e+03 true resid norm 4.522093337153e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.338757544659e-06 true resid norm 2.556045980853e-04 ||r(i)||/||b|| 5.652351223829e-07
>   3 SNES Function norm 4.042911112972e+02 
>     0 KSP preconditioned resid norm 7.996536035697e+02 true resid norm 4.042911112972e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 8.612178921208e-07 true resid norm 2.026756625099e-04 ||r(i)||/||b|| 5.013112008810e-07
>   4 SNES Function norm 3.618952927134e+02 
>     0 KSP preconditioned resid norm 6.392806337991e+02 true resid norm 3.618952927134e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.099112497936e-07 true resid norm 1.605851784448e-04 ||r(i)||/||b|| 4.437338138353e-07
>   5 SNES Function norm 3.239307814846e+02 
>     0 KSP preconditioned resid norm 5.205889419531e+02 true resid norm 3.239307814846e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.364948858968e-07 true resid norm 1.326652268287e-04 ||r(i)||/||b|| 4.095480714142e-07
>   6 SNES Function norm 2.899625386925e+02 
>     0 KSP preconditioned resid norm 4.305631057215e+02 true resid norm 2.899625386925e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.472775427751e-07 true resid norm 1.094956512355e-04 ||r(i)||/||b|| 3.776199909453e-07
>   7 SNES Function norm 2.596235386610e+02 
>     0 KSP preconditioned resid norm 3.606693697722e+02 true resid norm 2.596235386610e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.804499181789e-07 true resid norm 9.162646660567e-05 ||r(i)||/||b|| 3.529204904849e-07
>   8 SNES Function norm 2.178613867932e+02 
>     0 KSP preconditioned resid norm 2.791607776778e+02 true resid norm 2.178613867932e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.269331257925e-07 true resid norm 7.151934347788e-05 ||r(i)||/||b|| 3.282791160499e-07
>   9 SNES Function norm 1.541279494063e+02 
>     0 KSP preconditioned resid norm 1.781925534106e+02 true resid norm 1.541279494063e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.095285568305e-07 true resid norm 4.530373014757e-05 ||r(i)||/||b|| 2.939358521416e-07
>  10 SNES Function norm 1.361917300390e+02 
>     0 KSP preconditioned resid norm 4.801281559143e+01 true resid norm 1.361917300390e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.158004981151e-07 true resid norm 1.216364616255e-05 ||r(i)||/||b|| 8.931266354477e-08
>  11 SNES Function norm 6.951420638296e+00 
>     0 KSP preconditioned resid norm 2.808725633599e+00 true resid norm 6.951420638296e+00 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.361764665651e-09 true resid norm 7.088079744061e-07 ||r(i)||/||b|| 1.019659162188e-07
>  12 SNES Function norm 3.249486326694e-03 
>     0 KSP preconditioned resid norm 1.388995922430e-03 true resid norm 3.249486326694e-03 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.757348050151e-12 true resid norm 3.473453149942e-10 ||r(i)||/||b|| 1.068923762322e-07
>  13 SNES Function norm 4.590368792536e-05 
>     0 KSP preconditioned resid norm 4.360884570719e-06 true resid norm 4.590368792536e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.926789585311e-15 true resid norm 1.102338507065e-12 ||r(i)||/||b|| 2.401416001385e-08
>  14 SNES Function norm 4.422797479565e-05 
>     0 KSP preconditioned resid norm 5.831925614907e-06 true resid norm 4.422797479565e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.584144701532e-15 true resid norm 1.478006791605e-12 ||r(i)||/||b|| 3.341791701821e-08
>  15 SNES Function norm 4.371345326958e-05 
>     0 KSP preconditioned resid norm 4.897666437748e-06 true resid norm 4.371345326958e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.080247182926e-15 true resid norm 1.236587295819e-12 ||r(i)||/||b|| 2.828848336902e-08
>  16 SNES Function norm 4.348609553058e-05 
>     0 KSP preconditioned resid norm 3.738450099362e-06 true resid norm 4.348609553058e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.655406222015e-15 true resid norm 9.403440338690e-13 ||r(i)||/||b|| 2.162401619174e-08
>  17 SNES Function norm 4.324907311342e-05 
>     0 KSP preconditioned resid norm 5.296206632183e-06 true resid norm 4.324907311342e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.018283198612e-15 true resid norm 1.334862856428e-12 ||r(i)||/||b|| 3.086454252850e-08
>  18 SNES Function norm 4.324904992696e-05 
>     0 KSP preconditioned resid norm 5.289306428405e-06 true resid norm 4.324904992696e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.691985954854e-15 true resid norm 1.335680822478e-12 ||r(i)||/||b|| 3.088347200075e-08
>  19 SNES Function norm 4.324904992565e-05 
>     0 KSP preconditioned resid norm 5.289311834250e-06 true resid norm 4.324904992565e-05 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.455286406170e-15 true resid norm 1.337387995906e-12 ||r(i)||/||b|| 3.092294508677e-08
>  
> 
> At this point, it terminates with a diverged line search.  
> 
> 
> -gideon
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From ling.zou at inl.gov  Thu Sep 24 17:57:37 2015
From: ling.zou at inl.gov (Zou (Non-US), Ling)
Date: Thu, 24 Sep 2015 16:57:37 -0600
Subject: [petsc-users] SNES stuck with small residual
In-Reply-To: <A5A1A71A-9EF3-4E34-BC3B-9C1C789F30A0@mcs.anl.gov>
References: <DDE3BB2E-8CED-47B0-9BE8-9040666B0832@gmail.com>
	<CAMYG4GmQbUfUOCva_AfmWjTYB+LMgjeujzPX1g6XCNuewUw+8w@mail.gmail.com>
	<A5A1A71A-9EF3-4E34-BC3B-9C1C789F30A0@mcs.anl.gov>
Message-ID: <CAJWxZDTJC8CCy37thqLPH3rTppve2_n4RA3QMWdBsSTQK4fZqw@mail.gmail.com>

I would suggest try '-snes_mffd_type ds' if you are using Jacobian free.
It is a magic word that always helps me out of trouble.

Ling

On Thu, Sep 24, 2015 at 4:44 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   I completely disagree with Matt, looking at the numbers one sees very
> typical Newton behavior
>
> a little slow at the beginning because it is not in the quadratic basin
>
> 0 SNES Function norm 5.979113140046e+02
>   1 SNES Function norm 5.106334303399e+02
>   2 SNES Function norm 4.522093337153e+02
>   3 SNES Function norm 4.042911112972e+02
>   4 SNES Function norm 3.618952927134e+02
>   5 SNES Function norm 3.239307814846e+02
>   6 SNES Function norm 2.899625386925e+02
>   7 SNES Function norm 2.596235386610e+02
>   8 SNES Function norm 2.178613867932e+02
>   9 SNES Function norm 1.541279494063e+02
>  10 SNES Function norm 1.361917300390e+02
>
>  suddenly fast convergence as it gets in the quadratic basin and behaves
> like Newton should behave
>
>  11 SNES Function norm 6.951420638296e+00
>  12 SNES Function norm 3.249486326694e-03
>  13 SNES Function norm 4.590368792536e-05
>
> no more convergence as it "runs out of digits"
>
>  14 SNES Function norm 4.422797479565e-05
>  15 SNES Function norm 4.371345326958e-05
>  16 SNES Function norm 4.348609553058e-05
>  17 SNES Function norm 4.324907311342e-05
>  18 SNES Function norm 4.324904992696e-05
>  19 SNES Function norm 4.324904992565e-05
>
> The only difference between this run and most other runs of Newton (when
> it works in the standard way) is that it "runs out of digits" early (after
> only reducing the residual norm by 1e-7 when for super nice problems it can
> do like 1e-12).
>
> I don't think your Jacobian is crap, I don't think the Jacobian direction
> is crap and the Jacobian is not not terribly ill-conditioned since the
> direct solver reduces the residual norm by 1e-8.
>
> My recommendation when Newton "runs out of digits" is to try using quad
> precision. With PETSc you just need a recent version of the GNU compilers
> and the configure options --with-precision=__float128
> --download-f2cblaslapack   plus you need to make sure your code always uses
> PetscReal and PetscScalar instead of "double" or whatever and whenever you
> call elementary functions like sin() you call PetscSinReal() or
> PetscSinScalar() etc so that it uses the quad precision version (if you
> miss one of these you will never see the quad precision).
>
>   Barry
>
> What does it mean for Newton to "run out of digits"; this is not a
> technical term but I believe it is related to how accurate your function
> evaluations can be (I think some people call this noise) when expressed
> using floating point numbers. Perhaps it is something like "ill-conditioned
> functions". If the function evaluations are only valid for 8 digits then it
> is not reasonable to expect Newton (or any floating point algorithm) to
> squeeze the norm of the function values to 1e.-12.
>
>
>
>
>
> > On Sep 24, 2015, at 5:28 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Thu, Sep 24, 2015 at 5:23 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> > Just wondering if anyone has any suggestions for this scenario:
> >
> > I have seen this happen a bunch. Your Newton directions are crap. That
> means
> >
> >   a) you have a crap Jacobian
> >
> >   b) your Jacobian direction is crap (unlikely here)
> >
> >   c) your Jacobian is very ill-conditioned
> >
> > Something to try in order to see what is happening is to start doing
> Picard when Newton stalls. If it
> > converges, then you have something like c) I think. We can try to
> mitigate this by NPC. Something
> > as simple as Newton + NCG or Newton + NRichardson can work.
> >
> >   Matt
> >
> >   0 SNES Function norm 5.979113140046e+02
> >     0 KSP preconditioned resid norm 1.609660749629e+03 true resid norm
> 5.979113140046e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.083640787055e-06 true resid norm
> 4.070012556358e-04 ||r(i)||/||b|| 6.807050579289e-07
> >   1 SNES Function norm 5.106334303399e+02
> >     0 KSP preconditioned resid norm 1.304622870569e+03 true resid norm
> 5.106334303399e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.542022975926e-06 true resid norm
> 3.284828506619e-04 ||r(i)||/||b|| 6.432850478341e-07
> >   2 SNES Function norm 4.522093337153e+02
> >     0 KSP preconditioned resid norm 1.017788935768e+03 true resid norm
> 4.522093337153e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.338757544659e-06 true resid norm
> 2.556045980853e-04 ||r(i)||/||b|| 5.652351223829e-07
> >   3 SNES Function norm 4.042911112972e+02
> >     0 KSP preconditioned resid norm 7.996536035697e+02 true resid norm
> 4.042911112972e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 8.612178921208e-07 true resid norm
> 2.026756625099e-04 ||r(i)||/||b|| 5.013112008810e-07
> >   4 SNES Function norm 3.618952927134e+02
> >     0 KSP preconditioned resid norm 6.392806337991e+02 true resid norm
> 3.618952927134e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.099112497936e-07 true resid norm
> 1.605851784448e-04 ||r(i)||/||b|| 4.437338138353e-07
> >   5 SNES Function norm 3.239307814846e+02
> >     0 KSP preconditioned resid norm 5.205889419531e+02 true resid norm
> 3.239307814846e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 3.364948858968e-07 true resid norm
> 1.326652268287e-04 ||r(i)||/||b|| 4.095480714142e-07
> >   6 SNES Function norm 2.899625386925e+02
> >     0 KSP preconditioned resid norm 4.305631057215e+02 true resid norm
> 2.899625386925e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 2.472775427751e-07 true resid norm
> 1.094956512355e-04 ||r(i)||/||b|| 3.776199909453e-07
> >   7 SNES Function norm 2.596235386610e+02
> >     0 KSP preconditioned resid norm 3.606693697722e+02 true resid norm
> 2.596235386610e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.804499181789e-07 true resid norm
> 9.162646660567e-05 ||r(i)||/||b|| 3.529204904849e-07
> >   8 SNES Function norm 2.178613867932e+02
> >     0 KSP preconditioned resid norm 2.791607776778e+02 true resid norm
> 2.178613867932e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.269331257925e-07 true resid norm
> 7.151934347788e-05 ||r(i)||/||b|| 3.282791160499e-07
> >   9 SNES Function norm 1.541279494063e+02
> >     0 KSP preconditioned resid norm 1.781925534106e+02 true resid norm
> 1.541279494063e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.095285568305e-07 true resid norm
> 4.530373014757e-05 ||r(i)||/||b|| 2.939358521416e-07
> >  10 SNES Function norm 1.361917300390e+02
> >     0 KSP preconditioned resid norm 4.801281559143e+01 true resid norm
> 1.361917300390e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.158004981151e-07 true resid norm
> 1.216364616255e-05 ||r(i)||/||b|| 8.931266354477e-08
> >  11 SNES Function norm 6.951420638296e+00
> >     0 KSP preconditioned resid norm 2.808725633599e+00 true resid norm
> 6.951420638296e+00 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 2.361764665651e-09 true resid norm
> 7.088079744061e-07 ||r(i)||/||b|| 1.019659162188e-07
> >  12 SNES Function norm 3.249486326694e-03
> >     0 KSP preconditioned resid norm 1.388995922430e-03 true resid norm
> 3.249486326694e-03 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.757348050151e-12 true resid norm
> 3.473453149942e-10 ||r(i)||/||b|| 1.068923762322e-07
> >  13 SNES Function norm 4.590368792536e-05
> >     0 KSP preconditioned resid norm 4.360884570719e-06 true resid norm
> 4.590368792536e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 3.926789585311e-15 true resid norm
> 1.102338507065e-12 ||r(i)||/||b|| 2.401416001385e-08
> >  14 SNES Function norm 4.422797479565e-05
> >     0 KSP preconditioned resid norm 5.831925614907e-06 true resid norm
> 4.422797479565e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.584144701532e-15 true resid norm
> 1.478006791605e-12 ||r(i)||/||b|| 3.341791701821e-08
> >  15 SNES Function norm 4.371345326958e-05
> >     0 KSP preconditioned resid norm 4.897666437748e-06 true resid norm
> 4.371345326958e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 4.080247182926e-15 true resid norm
> 1.236587295819e-12 ||r(i)||/||b|| 2.828848336902e-08
> >  16 SNES Function norm 4.348609553058e-05
> >     0 KSP preconditioned resid norm 3.738450099362e-06 true resid norm
> 4.348609553058e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 3.655406222015e-15 true resid norm
> 9.403440338690e-13 ||r(i)||/||b|| 2.162401619174e-08
> >  17 SNES Function norm 4.324907311342e-05
> >     0 KSP preconditioned resid norm 5.296206632183e-06 true resid norm
> 4.324907311342e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.018283198612e-15 true resid norm
> 1.334862856428e-12 ||r(i)||/||b|| 3.086454252850e-08
> >  18 SNES Function norm 4.324904992696e-05
> >     0 KSP preconditioned resid norm 5.289306428405e-06 true resid norm
> 4.324904992696e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.691985954854e-15 true resid norm
> 1.335680822478e-12 ||r(i)||/||b|| 3.088347200075e-08
> >  19 SNES Function norm 4.324904992565e-05
> >     0 KSP preconditioned resid norm 5.289311834250e-06 true resid norm
> 4.324904992565e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.455286406170e-15 true resid norm
> 1.337387995906e-12 ||r(i)||/||b|| 3.092294508677e-08
> >
> >
> > At this point, it terminates with a diverged line search.
> >
> >
> > -gideon
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/c65bbbf1/attachment.html>

From knepley at gmail.com  Thu Sep 24 18:01:29 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 24 Sep 2015 18:01:29 -0500
Subject: [petsc-users] SNES stuck with small residual
In-Reply-To: <A5A1A71A-9EF3-4E34-BC3B-9C1C789F30A0@mcs.anl.gov>
References: <DDE3BB2E-8CED-47B0-9BE8-9040666B0832@gmail.com>
	<CAMYG4GmQbUfUOCva_AfmWjTYB+LMgjeujzPX1g6XCNuewUw+8w@mail.gmail.com>
	<A5A1A71A-9EF3-4E34-BC3B-9C1C789F30A0@mcs.anl.gov>
Message-ID: <CAMYG4GkhDoy=+L3aubXTgdE-pZ7h01AG4bP77RKhYBwUatpQ8A@mail.gmail.com>

On Thu, Sep 24, 2015 at 5:44 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   I completely disagree with Matt, looking at the numbers one sees very
> typical Newton behavior
>
> a little slow at the beginning because it is not in the quadratic basin
>
> 0 SNES Function norm 5.979113140046e+02
>   1 SNES Function norm 5.106334303399e+02
>   2 SNES Function norm 4.522093337153e+02
>   3 SNES Function norm 4.042911112972e+02
>   4 SNES Function norm 3.618952927134e+02
>   5 SNES Function norm 3.239307814846e+02
>   6 SNES Function norm 2.899625386925e+02
>   7 SNES Function norm 2.596235386610e+02
>   8 SNES Function norm 2.178613867932e+02
>   9 SNES Function norm 1.541279494063e+02
>  10 SNES Function norm 1.361917300390e+02
>
>  suddenly fast convergence as it gets in the quadratic basin and behaves
> like Newton should behave
>
>  11 SNES Function norm 6.951420638296e+00
>  12 SNES Function norm 3.249486326694e-03
>  13 SNES Function norm 4.590368792536e-05
>

Not exactly quadratic, but definitely faster than beforeIt still looks
about linear with c = 0.01. However,
Barry might be right since this rapid convergence arrests suddenly.

Can you think of a reason why your function evaluation would only be single
precision?

   Matt


> no more convergence as it "runs out of digits"
>
>  14 SNES Function norm 4.422797479565e-05
>  15 SNES Function norm 4.371345326958e-05
>  16 SNES Function norm 4.348609553058e-05
>  17 SNES Function norm 4.324907311342e-05
>  18 SNES Function norm 4.324904992696e-05
>  19 SNES Function norm 4.324904992565e-05
>
> The only difference between this run and most other runs of Newton (when
> it works in the standard way) is that it "runs out of digits" early (after
> only reducing the residual norm by 1e-7 when for super nice problems it can
> do like 1e-12).
>
> I don't think your Jacobian is crap, I don't think the Jacobian direction
> is crap and the Jacobian is not not terribly ill-conditioned since the
> direct solver reduces the residual norm by 1e-8.
>
> My recommendation when Newton "runs out of digits" is to try using quad
> precision. With PETSc you just need a recent version of the GNU compilers
> and the configure options --with-precision=__float128
> --download-f2cblaslapack   plus you need to make sure your code always uses
> PetscReal and PetscScalar instead of "double" or whatever and whenever you
> call elementary functions like sin() you call PetscSinReal() or
> PetscSinScalar() etc so that it uses the quad precision version (if you
> miss one of these you will never see the quad precision).
>
>   Barry
>
> What does it mean for Newton to "run out of digits"; this is not a
> technical term but I believe it is related to how accurate your function
> evaluations can be (I think some people call this noise) when expressed
> using floating point numbers. Perhaps it is something like "ill-conditioned
> functions". If the function evaluations are only valid for 8 digits then it
> is not reasonable to expect Newton (or any floating point algorithm) to
> squeeze the norm of the function values to 1e.-12.
>
>
>
>
>
> > On Sep 24, 2015, at 5:28 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Thu, Sep 24, 2015 at 5:23 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> > Just wondering if anyone has any suggestions for this scenario:
> >
> > I have seen this happen a bunch. Your Newton directions are crap. That
> means
> >
> >   a) you have a crap Jacobian
> >
> >   b) your Jacobian direction is crap (unlikely here)
> >
> >   c) your Jacobian is very ill-conditioned
> >
> > Something to try in order to see what is happening is to start doing
> Picard when Newton stalls. If it
> > converges, then you have something like c) I think. We can try to
> mitigate this by NPC. Something
> > as simple as Newton + NCG or Newton + NRichardson can work.
> >
> >   Matt
> >
> >   0 SNES Function norm 5.979113140046e+02
> >     0 KSP preconditioned resid norm 1.609660749629e+03 true resid norm
> 5.979113140046e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.083640787055e-06 true resid norm
> 4.070012556358e-04 ||r(i)||/||b|| 6.807050579289e-07
> >   1 SNES Function norm 5.106334303399e+02
> >     0 KSP preconditioned resid norm 1.304622870569e+03 true resid norm
> 5.106334303399e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.542022975926e-06 true resid norm
> 3.284828506619e-04 ||r(i)||/||b|| 6.432850478341e-07
> >   2 SNES Function norm 4.522093337153e+02
> >     0 KSP preconditioned resid norm 1.017788935768e+03 true resid norm
> 4.522093337153e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.338757544659e-06 true resid norm
> 2.556045980853e-04 ||r(i)||/||b|| 5.652351223829e-07
> >   3 SNES Function norm 4.042911112972e+02
> >     0 KSP preconditioned resid norm 7.996536035697e+02 true resid norm
> 4.042911112972e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 8.612178921208e-07 true resid norm
> 2.026756625099e-04 ||r(i)||/||b|| 5.013112008810e-07
> >   4 SNES Function norm 3.618952927134e+02
> >     0 KSP preconditioned resid norm 6.392806337991e+02 true resid norm
> 3.618952927134e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.099112497936e-07 true resid norm
> 1.605851784448e-04 ||r(i)||/||b|| 4.437338138353e-07
> >   5 SNES Function norm 3.239307814846e+02
> >     0 KSP preconditioned resid norm 5.205889419531e+02 true resid norm
> 3.239307814846e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 3.364948858968e-07 true resid norm
> 1.326652268287e-04 ||r(i)||/||b|| 4.095480714142e-07
> >   6 SNES Function norm 2.899625386925e+02
> >     0 KSP preconditioned resid norm 4.305631057215e+02 true resid norm
> 2.899625386925e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 2.472775427751e-07 true resid norm
> 1.094956512355e-04 ||r(i)||/||b|| 3.776199909453e-07
> >   7 SNES Function norm 2.596235386610e+02
> >     0 KSP preconditioned resid norm 3.606693697722e+02 true resid norm
> 2.596235386610e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.804499181789e-07 true resid norm
> 9.162646660567e-05 ||r(i)||/||b|| 3.529204904849e-07
> >   8 SNES Function norm 2.178613867932e+02
> >     0 KSP preconditioned resid norm 2.791607776778e+02 true resid norm
> 2.178613867932e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.269331257925e-07 true resid norm
> 7.151934347788e-05 ||r(i)||/||b|| 3.282791160499e-07
> >   9 SNES Function norm 1.541279494063e+02
> >     0 KSP preconditioned resid norm 1.781925534106e+02 true resid norm
> 1.541279494063e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.095285568305e-07 true resid norm
> 4.530373014757e-05 ||r(i)||/||b|| 2.939358521416e-07
> >  10 SNES Function norm 1.361917300390e+02
> >     0 KSP preconditioned resid norm 4.801281559143e+01 true resid norm
> 1.361917300390e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.158004981151e-07 true resid norm
> 1.216364616255e-05 ||r(i)||/||b|| 8.931266354477e-08
> >  11 SNES Function norm 6.951420638296e+00
> >     0 KSP preconditioned resid norm 2.808725633599e+00 true resid norm
> 6.951420638296e+00 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 2.361764665651e-09 true resid norm
> 7.088079744061e-07 ||r(i)||/||b|| 1.019659162188e-07
> >  12 SNES Function norm 3.249486326694e-03
> >     0 KSP preconditioned resid norm 1.388995922430e-03 true resid norm
> 3.249486326694e-03 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.757348050151e-12 true resid norm
> 3.473453149942e-10 ||r(i)||/||b|| 1.068923762322e-07
> >  13 SNES Function norm 4.590368792536e-05
> >     0 KSP preconditioned resid norm 4.360884570719e-06 true resid norm
> 4.590368792536e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 3.926789585311e-15 true resid norm
> 1.102338507065e-12 ||r(i)||/||b|| 2.401416001385e-08
> >  14 SNES Function norm 4.422797479565e-05
> >     0 KSP preconditioned resid norm 5.831925614907e-06 true resid norm
> 4.422797479565e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.584144701532e-15 true resid norm
> 1.478006791605e-12 ||r(i)||/||b|| 3.341791701821e-08
> >  15 SNES Function norm 4.371345326958e-05
> >     0 KSP preconditioned resid norm 4.897666437748e-06 true resid norm
> 4.371345326958e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 4.080247182926e-15 true resid norm
> 1.236587295819e-12 ||r(i)||/||b|| 2.828848336902e-08
> >  16 SNES Function norm 4.348609553058e-05
> >     0 KSP preconditioned resid norm 3.738450099362e-06 true resid norm
> 4.348609553058e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 3.655406222015e-15 true resid norm
> 9.403440338690e-13 ||r(i)||/||b|| 2.162401619174e-08
> >  17 SNES Function norm 4.324907311342e-05
> >     0 KSP preconditioned resid norm 5.296206632183e-06 true resid norm
> 4.324907311342e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.018283198612e-15 true resid norm
> 1.334862856428e-12 ||r(i)||/||b|| 3.086454252850e-08
> >  18 SNES Function norm 4.324904992696e-05
> >     0 KSP preconditioned resid norm 5.289306428405e-06 true resid norm
> 4.324904992696e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.691985954854e-15 true resid norm
> 1.335680822478e-12 ||r(i)||/||b|| 3.088347200075e-08
> >  19 SNES Function norm 4.324904992565e-05
> >     0 KSP preconditioned resid norm 5.289311834250e-06 true resid norm
> 4.324904992565e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.455286406170e-15 true resid norm
> 1.337387995906e-12 ||r(i)||/||b|| 3.092294508677e-08
> >
> >
> > At this point, it terminates with a diverged line search.
> >
> >
> > -gideon
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/3663aa87/attachment-0001.html>

From gideon.simpson at gmail.com  Thu Sep 24 18:05:47 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Thu, 24 Sep 2015 19:05:47 -0400
Subject: [petsc-users] SNES stuck with small residual
In-Reply-To: <CAMYG4GkhDoy=+L3aubXTgdE-pZ7h01AG4bP77RKhYBwUatpQ8A@mail.gmail.com>
References: <DDE3BB2E-8CED-47B0-9BE8-9040666B0832@gmail.com>
	<CAMYG4GmQbUfUOCva_AfmWjTYB+LMgjeujzPX1g6XCNuewUw+8w@mail.gmail.com>
	<A5A1A71A-9EF3-4E34-BC3B-9C1C789F30A0@mcs.anl.gov>
	<CAMYG4GkhDoy=+L3aubXTgdE-pZ7h01AG4bP77RKhYBwUatpQ8A@mail.gmail.com>
Message-ID: <09A9EE66-D486-46A7-8D9C-ED37E7413363@gmail.com>

Nothing springs to mind in terms of the computation.  The only non-trivial calculation is the use of PetscPowScalar; everything else is straightforward +/-*.  The problem does have 10^7 unknowns; could that be an issue?

-gideon

> On Sep 24, 2015, at 7:01 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Thu, Sep 24, 2015 at 5:44 PM, Barry Smith <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
> 
>   I completely disagree with Matt, looking at the numbers one sees very typical Newton behavior
> 
> a little slow at the beginning because it is not in the quadratic basin
> 
> 0 SNES Function norm 5.979113140046e+02
>   1 SNES Function norm 5.106334303399e+02
>   2 SNES Function norm 4.522093337153e+02
>   3 SNES Function norm 4.042911112972e+02
>   4 SNES Function norm 3.618952927134e+02
>   5 SNES Function norm 3.239307814846e+02
>   6 SNES Function norm 2.899625386925e+02
>   7 SNES Function norm 2.596235386610e+02
>   8 SNES Function norm 2.178613867932e+02
>   9 SNES Function norm 1.541279494063e+02
>  10 SNES Function norm 1.361917300390e+02
> 
>  suddenly fast convergence as it gets in the quadratic basin and behaves like Newton should behave
> 
>  11 SNES Function norm 6.951420638296e+00
>  12 SNES Function norm 3.249486326694e-03
>  13 SNES Function norm 4.590368792536e-05
> 
> Not exactly quadratic, but definitely faster than beforeIt still looks about linear with c = 0.01. However,
> Barry might be right since this rapid convergence arrests suddenly.
> 
> Can you think of a reason why your function evaluation would only be single precision?
> 
>    Matt
>  
> no more convergence as it "runs out of digits"
> 
>  14 SNES Function norm 4.422797479565e-05
>  15 SNES Function norm 4.371345326958e-05
>  16 SNES Function norm 4.348609553058e-05
>  17 SNES Function norm 4.324907311342e-05
>  18 SNES Function norm 4.324904992696e-05
>  19 SNES Function norm 4.324904992565e-05
> 
> The only difference between this run and most other runs of Newton (when it works in the standard way) is that it "runs out of digits" early (after only reducing the residual norm by 1e-7 when for super nice problems it can do like 1e-12).
> 
> I don't think your Jacobian is crap, I don't think the Jacobian direction is crap and the Jacobian is not not terribly ill-conditioned since the direct solver reduces the residual norm by 1e-8.
> 
> My recommendation when Newton "runs out of digits" is to try using quad precision. With PETSc you just need a recent version of the GNU compilers and the configure options --with-precision=__float128 --download-f2cblaslapack   plus you need to make sure your code always uses PetscReal and PetscScalar instead of "double" or whatever and whenever you call elementary functions like sin() you call PetscSinReal() or PetscSinScalar() etc so that it uses the quad precision version (if you miss one of these you will never see the quad precision).
> 
>   Barry
> 
> What does it mean for Newton to "run out of digits"; this is not a technical term but I believe it is related to how accurate your function evaluations can be (I think some people call this noise) when expressed using floating point numbers. Perhaps it is something like "ill-conditioned functions". If the function evaluations are only valid for 8 digits then it is not reasonable to expect Newton (or any floating point algorithm) to squeeze the norm of the function values to 1e.-12.
> 
> 
> 
> 
> 
> > On Sep 24, 2015, at 5:28 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
> >
> > On Thu, Sep 24, 2015 at 5:23 PM, Gideon Simpson <gideon.simpson at gmail.com <mailto:gideon.simpson at gmail.com>> wrote:
> > Just wondering if anyone has any suggestions for this scenario:
> >
> > I have seen this happen a bunch. Your Newton directions are crap. That means
> >
> >   a) you have a crap Jacobian
> >
> >   b) your Jacobian direction is crap (unlikely here)
> >
> >   c) your Jacobian is very ill-conditioned
> >
> > Something to try in order to see what is happening is to start doing Picard when Newton stalls. If it
> > converges, then you have something like c) I think. We can try to mitigate this by NPC. Something
> > as simple as Newton + NCG or Newton + NRichardson can work.
> >
> >   Matt
> >
> >   0 SNES Function norm 5.979113140046e+02
> >     0 KSP preconditioned resid norm 1.609660749629e+03 true resid norm 5.979113140046e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.083640787055e-06 true resid norm 4.070012556358e-04 ||r(i)||/||b|| 6.807050579289e-07
> >   1 SNES Function norm 5.106334303399e+02
> >     0 KSP preconditioned resid norm 1.304622870569e+03 true resid norm 5.106334303399e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.542022975926e-06 true resid norm 3.284828506619e-04 ||r(i)||/||b|| 6.432850478341e-07
> >   2 SNES Function norm 4.522093337153e+02
> >     0 KSP preconditioned resid norm 1.017788935768e+03 true resid norm 4.522093337153e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.338757544659e-06 true resid norm 2.556045980853e-04 ||r(i)||/||b|| 5.652351223829e-07
> >   3 SNES Function norm 4.042911112972e+02
> >     0 KSP preconditioned resid norm 7.996536035697e+02 true resid norm 4.042911112972e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 8.612178921208e-07 true resid norm 2.026756625099e-04 ||r(i)||/||b|| 5.013112008810e-07
> >   4 SNES Function norm 3.618952927134e+02
> >     0 KSP preconditioned resid norm 6.392806337991e+02 true resid norm 3.618952927134e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.099112497936e-07 true resid norm 1.605851784448e-04 ||r(i)||/||b|| 4.437338138353e-07
> >   5 SNES Function norm 3.239307814846e+02
> >     0 KSP preconditioned resid norm 5.205889419531e+02 true resid norm 3.239307814846e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 3.364948858968e-07 true resid norm 1.326652268287e-04 ||r(i)||/||b|| 4.095480714142e-07
> >   6 SNES Function norm 2.899625386925e+02
> >     0 KSP preconditioned resid norm 4.305631057215e+02 true resid norm 2.899625386925e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 2.472775427751e-07 true resid norm 1.094956512355e-04 ||r(i)||/||b|| 3.776199909453e-07
> >   7 SNES Function norm 2.596235386610e+02
> >     0 KSP preconditioned resid norm 3.606693697722e+02 true resid norm 2.596235386610e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.804499181789e-07 true resid norm 9.162646660567e-05 ||r(i)||/||b|| 3.529204904849e-07
> >   8 SNES Function norm 2.178613867932e+02
> >     0 KSP preconditioned resid norm 2.791607776778e+02 true resid norm 2.178613867932e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.269331257925e-07 true resid norm 7.151934347788e-05 ||r(i)||/||b|| 3.282791160499e-07
> >   9 SNES Function norm 1.541279494063e+02
> >     0 KSP preconditioned resid norm 1.781925534106e+02 true resid norm 1.541279494063e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.095285568305e-07 true resid norm 4.530373014757e-05 ||r(i)||/||b|| 2.939358521416e-07
> >  10 SNES Function norm 1.361917300390e+02
> >     0 KSP preconditioned resid norm 4.801281559143e+01 true resid norm 1.361917300390e+02 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.158004981151e-07 true resid norm 1.216364616255e-05 ||r(i)||/||b|| 8.931266354477e-08
> >  11 SNES Function norm 6.951420638296e+00
> >     0 KSP preconditioned resid norm 2.808725633599e+00 true resid norm 6.951420638296e+00 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 2.361764665651e-09 true resid norm 7.088079744061e-07 ||r(i)||/||b|| 1.019659162188e-07
> >  12 SNES Function norm 3.249486326694e-03
> >     0 KSP preconditioned resid norm 1.388995922430e-03 true resid norm 3.249486326694e-03 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 1.757348050151e-12 true resid norm 3.473453149942e-10 ||r(i)||/||b|| 1.068923762322e-07
> >  13 SNES Function norm 4.590368792536e-05
> >     0 KSP preconditioned resid norm 4.360884570719e-06 true resid norm 4.590368792536e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 3.926789585311e-15 true resid norm 1.102338507065e-12 ||r(i)||/||b|| 2.401416001385e-08
> >  14 SNES Function norm 4.422797479565e-05
> >     0 KSP preconditioned resid norm 5.831925614907e-06 true resid norm 4.422797479565e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.584144701532e-15 true resid norm 1.478006791605e-12 ||r(i)||/||b|| 3.341791701821e-08
> >  15 SNES Function norm 4.371345326958e-05
> >     0 KSP preconditioned resid norm 4.897666437748e-06 true resid norm 4.371345326958e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 4.080247182926e-15 true resid norm 1.236587295819e-12 ||r(i)||/||b|| 2.828848336902e-08
> >  16 SNES Function norm 4.348609553058e-05
> >     0 KSP preconditioned resid norm 3.738450099362e-06 true resid norm 4.348609553058e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 3.655406222015e-15 true resid norm 9.403440338690e-13 ||r(i)||/||b|| 2.162401619174e-08
> >  17 SNES Function norm 4.324907311342e-05
> >     0 KSP preconditioned resid norm 5.296206632183e-06 true resid norm 4.324907311342e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.018283198612e-15 true resid norm 1.334862856428e-12 ||r(i)||/||b|| 3.086454252850e-08
> >  18 SNES Function norm 4.324904992696e-05
> >     0 KSP preconditioned resid norm 5.289306428405e-06 true resid norm 4.324904992696e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.691985954854e-15 true resid norm 1.335680822478e-12 ||r(i)||/||b|| 3.088347200075e-08
> >  19 SNES Function norm 4.324904992565e-05
> >     0 KSP preconditioned resid norm 5.289311834250e-06 true resid norm 4.324904992565e-05 ||r(i)||/||b|| 1.000000000000e+00
> >     1 KSP preconditioned resid norm 5.455286406170e-15 true resid norm 1.337387995906e-12 ||r(i)||/||b|| 3.092294508677e-08
> >
> >
> > At this point, it terminates with a diverged line search.
> >
> >
> > -gideon
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/91698339/attachment.html>

From knepley at gmail.com  Thu Sep 24 18:27:12 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 24 Sep 2015 18:27:12 -0500
Subject: [petsc-users] SNES stuck with small residual
In-Reply-To: <09A9EE66-D486-46A7-8D9C-ED37E7413363@gmail.com>
References: <DDE3BB2E-8CED-47B0-9BE8-9040666B0832@gmail.com>
	<CAMYG4GmQbUfUOCva_AfmWjTYB+LMgjeujzPX1g6XCNuewUw+8w@mail.gmail.com>
	<A5A1A71A-9EF3-4E34-BC3B-9C1C789F30A0@mcs.anl.gov>
	<CAMYG4GkhDoy=+L3aubXTgdE-pZ7h01AG4bP77RKhYBwUatpQ8A@mail.gmail.com>
	<09A9EE66-D486-46A7-8D9C-ED37E7413363@gmail.com>
Message-ID: <CAMYG4Gk7UZkKVJc=NNGjEOmy1X+1GhX8arA74hYXOQbdHjShrQ@mail.gmail.com>

On Thu, Sep 24, 2015 at 6:05 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Nothing springs to mind in terms of the computation.  The only non-trivial
> calculation is the use of PetscPowScalar; everything else is
> straightforward +/-*.  The problem does have 10^7 unknowns; could that be
> an issue?
>

The unknowns alone would not cause this. You could have catastrophic
cancellation in your evaluation, even just using -. I am
suspicious of this explanation, but Barry is right that is can be tested
fairly easily by using 128.

   Matt


> -gideon
>
> On Sep 24, 2015, at 7:01 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Thu, Sep 24, 2015 at 5:44 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   I completely disagree with Matt, looking at the numbers one sees very
>> typical Newton behavior
>>
>> a little slow at the beginning because it is not in the quadratic basin
>>
>> 0 SNES Function norm 5.979113140046e+02
>>   1 SNES Function norm 5.106334303399e+02
>>   2 SNES Function norm 4.522093337153e+02
>>   3 SNES Function norm 4.042911112972e+02
>>   4 SNES Function norm 3.618952927134e+02
>>   5 SNES Function norm 3.239307814846e+02
>>   6 SNES Function norm 2.899625386925e+02
>>   7 SNES Function norm 2.596235386610e+02
>>   8 SNES Function norm 2.178613867932e+02
>>   9 SNES Function norm 1.541279494063e+02
>>  10 SNES Function norm 1.361917300390e+02
>>
>>  suddenly fast convergence as it gets in the quadratic basin and behaves
>> like Newton should behave
>>
>>  11 SNES Function norm 6.951420638296e+00
>>  12 SNES Function norm 3.249486326694e-03
>>  13 SNES Function norm 4.590368792536e-05
>>
>
> Not exactly quadratic, but definitely faster than beforeIt still looks
> about linear with c = 0.01. However,
> Barry might be right since this rapid convergence arrests suddenly.
>
> Can you think of a reason why your function evaluation would only be
> single precision?
>
>    Matt
>
>
>> no more convergence as it "runs out of digits"
>>
>>  14 SNES Function norm 4.422797479565e-05
>>  15 SNES Function norm 4.371345326958e-05
>>  16 SNES Function norm 4.348609553058e-05
>>  17 SNES Function norm 4.324907311342e-05
>>  18 SNES Function norm 4.324904992696e-05
>>  19 SNES Function norm 4.324904992565e-05
>>
>> The only difference between this run and most other runs of Newton (when
>> it works in the standard way) is that it "runs out of digits" early (after
>> only reducing the residual norm by 1e-7 when for super nice problems it can
>> do like 1e-12).
>>
>> I don't think your Jacobian is crap, I don't think the Jacobian direction
>> is crap and the Jacobian is not not terribly ill-conditioned since the
>> direct solver reduces the residual norm by 1e-8.
>>
>> My recommendation when Newton "runs out of digits" is to try using quad
>> precision. With PETSc you just need a recent version of the GNU compilers
>> and the configure options --with-precision=__float128
>> --download-f2cblaslapack   plus you need to make sure your code always uses
>> PetscReal and PetscScalar instead of "double" or whatever and whenever you
>> call elementary functions like sin() you call PetscSinReal() or
>> PetscSinScalar() etc so that it uses the quad precision version (if you
>> miss one of these you will never see the quad precision).
>>
>>   Barry
>>
>> What does it mean for Newton to "run out of digits"; this is not a
>> technical term but I believe it is related to how accurate your function
>> evaluations can be (I think some people call this noise) when expressed
>> using floating point numbers. Perhaps it is something like "ill-conditioned
>> functions". If the function evaluations are only valid for 8 digits then it
>> is not reasonable to expect Newton (or any floating point algorithm) to
>> squeeze the norm of the function values to 1e.-12.
>>
>>
>>
>>
>>
>> > On Sep 24, 2015, at 5:28 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> >
>> > On Thu, Sep 24, 2015 at 5:23 PM, Gideon Simpson <
>> gideon.simpson at gmail.com> wrote:
>> > Just wondering if anyone has any suggestions for this scenario:
>> >
>> > I have seen this happen a bunch. Your Newton directions are crap. That
>> means
>> >
>> >   a) you have a crap Jacobian
>> >
>> >   b) your Jacobian direction is crap (unlikely here)
>> >
>> >   c) your Jacobian is very ill-conditioned
>> >
>> > Something to try in order to see what is happening is to start doing
>> Picard when Newton stalls. If it
>> > converges, then you have something like c) I think. We can try to
>> mitigate this by NPC. Something
>> > as simple as Newton + NCG or Newton + NRichardson can work.
>> >
>> >   Matt
>> >
>> >   0 SNES Function norm 5.979113140046e+02
>> >     0 KSP preconditioned resid norm 1.609660749629e+03 true resid norm
>> 5.979113140046e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 1.083640787055e-06 true resid norm
>> 4.070012556358e-04 ||r(i)||/||b|| 6.807050579289e-07
>> >   1 SNES Function norm 5.106334303399e+02
>> >     0 KSP preconditioned resid norm 1.304622870569e+03 true resid norm
>> 5.106334303399e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 1.542022975926e-06 true resid norm
>> 3.284828506619e-04 ||r(i)||/||b|| 6.432850478341e-07
>> >   2 SNES Function norm 4.522093337153e+02
>> >     0 KSP preconditioned resid norm 1.017788935768e+03 true resid norm
>> 4.522093337153e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 1.338757544659e-06 true resid norm
>> 2.556045980853e-04 ||r(i)||/||b|| 5.652351223829e-07
>> >   3 SNES Function norm 4.042911112972e+02
>> >     0 KSP preconditioned resid norm 7.996536035697e+02 true resid norm
>> 4.042911112972e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 8.612178921208e-07 true resid norm
>> 2.026756625099e-04 ||r(i)||/||b|| 5.013112008810e-07
>> >   4 SNES Function norm 3.618952927134e+02
>> >     0 KSP preconditioned resid norm 6.392806337991e+02 true resid norm
>> 3.618952927134e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 5.099112497936e-07 true resid norm
>> 1.605851784448e-04 ||r(i)||/||b|| 4.437338138353e-07
>> >   5 SNES Function norm 3.239307814846e+02
>> >     0 KSP preconditioned resid norm 5.205889419531e+02 true resid norm
>> 3.239307814846e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 3.364948858968e-07 true resid norm
>> 1.326652268287e-04 ||r(i)||/||b|| 4.095480714142e-07
>> >   6 SNES Function norm 2.899625386925e+02
>> >     0 KSP preconditioned resid norm 4.305631057215e+02 true resid norm
>> 2.899625386925e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 2.472775427751e-07 true resid norm
>> 1.094956512355e-04 ||r(i)||/||b|| 3.776199909453e-07
>> >   7 SNES Function norm 2.596235386610e+02
>> >     0 KSP preconditioned resid norm 3.606693697722e+02 true resid norm
>> 2.596235386610e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 1.804499181789e-07 true resid norm
>> 9.162646660567e-05 ||r(i)||/||b|| 3.529204904849e-07
>> >   8 SNES Function norm 2.178613867932e+02
>> >     0 KSP preconditioned resid norm 2.791607776778e+02 true resid norm
>> 2.178613867932e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 1.269331257925e-07 true resid norm
>> 7.151934347788e-05 ||r(i)||/||b|| 3.282791160499e-07
>> >   9 SNES Function norm 1.541279494063e+02
>> >     0 KSP preconditioned resid norm 1.781925534106e+02 true resid norm
>> 1.541279494063e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 1.095285568305e-07 true resid norm
>> 4.530373014757e-05 ||r(i)||/||b|| 2.939358521416e-07
>> >  10 SNES Function norm 1.361917300390e+02
>> >     0 KSP preconditioned resid norm 4.801281559143e+01 true resid norm
>> 1.361917300390e+02 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 1.158004981151e-07 true resid norm
>> 1.216364616255e-05 ||r(i)||/||b|| 8.931266354477e-08
>> >  11 SNES Function norm 6.951420638296e+00
>> >     0 KSP preconditioned resid norm 2.808725633599e+00 true resid norm
>> 6.951420638296e+00 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 2.361764665651e-09 true resid norm
>> 7.088079744061e-07 ||r(i)||/||b|| 1.019659162188e-07
>> >  12 SNES Function norm 3.249486326694e-03
>> >     0 KSP preconditioned resid norm 1.388995922430e-03 true resid norm
>> 3.249486326694e-03 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 1.757348050151e-12 true resid norm
>> 3.473453149942e-10 ||r(i)||/||b|| 1.068923762322e-07
>> >  13 SNES Function norm 4.590368792536e-05
>> >     0 KSP preconditioned resid norm 4.360884570719e-06 true resid norm
>> 4.590368792536e-05 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 3.926789585311e-15 true resid norm
>> 1.102338507065e-12 ||r(i)||/||b|| 2.401416001385e-08
>> >  14 SNES Function norm 4.422797479565e-05
>> >     0 KSP preconditioned resid norm 5.831925614907e-06 true resid norm
>> 4.422797479565e-05 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 5.584144701532e-15 true resid norm
>> 1.478006791605e-12 ||r(i)||/||b|| 3.341791701821e-08
>> >  15 SNES Function norm 4.371345326958e-05
>> >     0 KSP preconditioned resid norm 4.897666437748e-06 true resid norm
>> 4.371345326958e-05 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 4.080247182926e-15 true resid norm
>> 1.236587295819e-12 ||r(i)||/||b|| 2.828848336902e-08
>> >  16 SNES Function norm 4.348609553058e-05
>> >     0 KSP preconditioned resid norm 3.738450099362e-06 true resid norm
>> 4.348609553058e-05 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 3.655406222015e-15 true resid norm
>> 9.403440338690e-13 ||r(i)||/||b|| 2.162401619174e-08
>> >  17 SNES Function norm 4.324907311342e-05
>> >     0 KSP preconditioned resid norm 5.296206632183e-06 true resid norm
>> 4.324907311342e-05 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 5.018283198612e-15 true resid norm
>> 1.334862856428e-12 ||r(i)||/||b|| 3.086454252850e-08
>> >  18 SNES Function norm 4.324904992696e-05
>> >     0 KSP preconditioned resid norm 5.289306428405e-06 true resid norm
>> 4.324904992696e-05 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 5.691985954854e-15 true resid norm
>> 1.335680822478e-12 ||r(i)||/||b|| 3.088347200075e-08
>> >  19 SNES Function norm 4.324904992565e-05
>> >     0 KSP preconditioned resid norm 5.289311834250e-06 true resid norm
>> 4.324904992565e-05 ||r(i)||/||b|| 1.000000000000e+00
>> >     1 KSP preconditioned resid norm 5.455286406170e-15 true resid norm
>> 1.337387995906e-12 ||r(i)||/||b|| 3.092294508677e-08
>> >
>> >
>> > At this point, it terminates with a diverged line search.
>> >
>> >
>> > -gideon
>> >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/35c4fef2/attachment-0001.html>

From valencia at astro.utoronto.ca  Thu Sep 24 19:21:36 2015
From: valencia at astro.utoronto.ca (Diana Valencia)
Date: Thu, 24 Sep 2015 20:21:36 -0400
Subject: [petsc-users] problem linking(?) a third party fortran code that
	uses petsc
Message-ID: <88D56551-9339-4AA1-B02D-A6EB41B1F690@astro.utoronto.ca>

Hi,


I am trying to compile a third party software written in fortran90 that uses petsc.  There is the option of compiling the code with or without petsc.  Without it I have no problems, however when I use petsc, I encounter this:

After building the subroutines the error is:

gfortran  -fdefault-real-8 -fdefault-double-8  -w -O2    -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o  blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o /opt/X11/lib/libpng.dylib   
Undefined symbols for architecture x86_64:
  "_kspcreate_", referenced from:
      ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
  "_kspdestroy_", referenced from:
      ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
  "_kspgetpc_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
  "_kspsetfromoptions_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
  "_kspsetoperators_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
  "_kspsettype_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
  "_kspsolve_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
  "_matassemblybegin_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
  "_matassemblyend_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
  "_matcreateseqaij_", referenced from:
      ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
  "_matdestroy_", referenced from:
      ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
  "_matsetvalues_", referenced from:
      ___petscimple_MOD_simplepetsc_insert_compactrow in PETScimple_mod.o
  "_mpi_comm_size_", referenced from:
      ___petscimple_MOD_initialize in PETScimple_mod.o
  "_pcfactorsetmatsolverpackage_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
  "_pcsettype_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
  "_petscfinalize_", referenced from:
      ___petscimple_MOD_finalize in PETScimple_mod.o
  "_petscinitialize_", referenced from:
      ___petscimple_MOD_initialize in PETScimple_mod.o
  "_veccreateseqwitharray_", referenced from:
      ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
  "_vecdestroy_", referenced from:
      ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
  "_vecplacearray_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
  "_vecresetarray_", referenced from:
      ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make: *** [stagyy] Error 1



My machine is OS X 10.10 and I am working with 64 bytes.  I configured petsc with the following flags:

./configure --with-fc=gfortran --with-fortran-datatypes --with-x=0 --download-mpich --download-parmetis --download-superlu --download-superlu_dist --download-blacs --download-scalapack --download-mumps --download-umfpack --download-ml --with-debugging=no --download-f-blas-lapack=yes

Any help is greatly appreciated!!
Thanks
Diana


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/197eddde/attachment.html>

From knepley at gmail.com  Thu Sep 24 19:34:19 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 24 Sep 2015 19:34:19 -0500
Subject: [petsc-users] problem linking(?) a third party fortran code
 that uses petsc
In-Reply-To: <88D56551-9339-4AA1-B02D-A6EB41B1F690@astro.utoronto.ca>
References: <88D56551-9339-4AA1-B02D-A6EB41B1F690@astro.utoronto.ca>
Message-ID: <CAMYG4G=T3WXoZQw44K5XsKRVuwkiAKwzLMGe25kgDmBjo97jMg@mail.gmail.com>

On Thu, Sep 24, 2015 at 7:21 PM, Diana Valencia <valencia at astro.utoronto.ca>
wrote:

> Hi,
>
>
> I am trying to compile a third party software written in fortran90 that
> uses petsc.  There is the option of compiling the code with or without
> petsc.  Without it I have no problems, however when I use petsc, I
> encounter this:
>
> After building the subroutines the error is:
>

Nowhere in this link line do I see -lpetsc. Are you sure you have this
build system configured correctly?

   Matt


> gfortran  -fdefault-real-8 -fdefault-double-8  -w -O2    -o stagyy main.o
> advdif.o communication.o continents.o evocore.o geometry.o init.o io.o
> iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o
> poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o
> residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o
> implicit_diffusion.o linpack.o plot.o wrtpng.o  blas123.o lapack.o
> nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o
> composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o
> io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o
> phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o
> timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o
> platesLea_mod.o /opt/X11/lib/libpng.dylib
> Undefined symbols for architecture x86_64:
>   "_kspcreate_", referenced from:
>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>   "_kspdestroy_", referenced from:
>       ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
>   "_kspgetpc_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_kspsetfromoptions_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_kspsetoperators_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_kspsettype_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_kspsolve_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_matassemblybegin_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_matassemblyend_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_matcreateseqaij_", referenced from:
>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>   "_matdestroy_", referenced from:
>       ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
>   "_matsetvalues_", referenced from:
>       ___petscimple_MOD_simplepetsc_insert_compactrow in PETScimple_mod.o
>   "_mpi_comm_size_", referenced from:
>       ___petscimple_MOD_initialize in PETScimple_mod.o
>   "_pcfactorsetmatsolverpackage_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_pcsettype_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_petscfinalize_", referenced from:
>       ___petscimple_MOD_finalize in PETScimple_mod.o
>   "_petscinitialize_", referenced from:
>       ___petscimple_MOD_initialize in PETScimple_mod.o
>   "_veccreateseqwitharray_", referenced from:
>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>   "_vecdestroy_", referenced from:
>       ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
>   "_vecplacearray_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_vecresetarray_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
> ld: symbol(s) not found for architecture x86_64
> collect2: error: ld returned 1 exit status
> make: *** [stagyy] Error 1
>
>
>
> My machine is OS X 10.10 and I am working with 64 bytes.  I configured
> petsc with the following flags:
>
> ./configure --with-fc=gfortran --with-fortran-datatypes --with-x=0
> --download-mpich --download-parmetis --download-superlu
> --download-superlu_dist --download-blacs --download-scalapack
> --download-mumps --download-umfpack --download-ml --with-debugging=no
> --download-f-blas-lapack=yes
>
> Any help is greatly appreciated!!
> Thanks
> Diana
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/53dc78e3/attachment.html>

From valencia at astro.utoronto.ca  Thu Sep 24 19:39:15 2015
From: valencia at astro.utoronto.ca (Diana Valencia)
Date: Thu, 24 Sep 2015 20:39:15 -0400
Subject: [petsc-users] problem linking(?) a third party fortran code
	that uses petsc
In-Reply-To: <CAMYG4G=T3WXoZQw44K5XsKRVuwkiAKwzLMGe25kgDmBjo97jMg@mail.gmail.com>
References: <88D56551-9339-4AA1-B02D-A6EB41B1F690@astro.utoronto.ca>
	<CAMYG4G=T3WXoZQw44K5XsKRVuwkiAKwzLMGe25kgDmBjo97jMg@mail.gmail.com>
Message-ID: <34F41684-B208-4028-B0C0-771EA817C390@astro.utoronto.ca>

Thanks for the response.  In compiling the subroutines, one that uses petsc is compiled successfully with:

gfortran  -fdefault-real-8 -fdefault-double-8  -w -O2    -I /Users/valencia/software/petsc/petsc-3.2-p7/arch-darwin-c-opt/include -c PETScimple_mod.f90

_______________________
Diana Valencia
Assistant Professor,  Physics & Astrophysics
Department of Physical and Environmental Sciences 
University of Toronto, Scarborough
1265 Military Trail, Toronto, ON, Canada, M1C 1A4
Voice 416 208-2986

> On Sep 24, 2015, at 8:34 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Thu, Sep 24, 2015 at 7:21 PM, Diana Valencia <valencia at astro.utoronto.ca <mailto:valencia at astro.utoronto.ca>> wrote:
> Hi,
> 
> 
> I am trying to compile a third party software written in fortran90 that uses petsc.  There is the option of compiling the code with or without petsc.  Without it I have no problems, however when I use petsc, I encounter this:
> 
> After building the subroutines the error is:
> 
> Nowhere in this link line do I see -lpetsc. Are you sure you have this build system configured correctly?
> 
>    Matt
>  
> gfortran  -fdefault-real-8 -fdefault-double-8  -w -O2    -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o  blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o /opt/X11/lib/libpng.dylib   
> Undefined symbols for architecture x86_64:
>   "_kspcreate_", referenced from:
>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>   "_kspdestroy_", referenced from:
>       ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
>   "_kspgetpc_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_kspsetfromoptions_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_kspsetoperators_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_kspsettype_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_kspsolve_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_matassemblybegin_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_matassemblyend_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_matcreateseqaij_", referenced from:
>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>   "_matdestroy_", referenced from:
>       ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
>   "_matsetvalues_", referenced from:
>       ___petscimple_MOD_simplepetsc_insert_compactrow in PETScimple_mod.o
>   "_mpi_comm_size_", referenced from:
>       ___petscimple_MOD_initialize in PETScimple_mod.o
>   "_pcfactorsetmatsolverpackage_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_pcsettype_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_petscfinalize_", referenced from:
>       ___petscimple_MOD_finalize in PETScimple_mod.o
>   "_petscinitialize_", referenced from:
>       ___petscimple_MOD_initialize in PETScimple_mod.o
>   "_veccreateseqwitharray_", referenced from:
>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>   "_vecdestroy_", referenced from:
>       ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
>   "_vecplacearray_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>   "_vecresetarray_", referenced from:
>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
> ld: symbol(s) not found for architecture x86_64
> collect2: error: ld returned 1 exit status
> make: *** [stagyy] Error 1
> 
> 
> 
> My machine is OS X 10.10 and I am working with 64 bytes.  I configured petsc with the following flags:
> 
> ./configure --with-fc=gfortran --with-fortran-datatypes --with-x=0 --download-mpich --download-parmetis --download-superlu --download-superlu_dist --download-blacs --download-scalapack --download-mumps --download-umfpack --download-ml --with-debugging=no --download-f-blas-lapack=yes
> 
> Any help is greatly appreciated!!
> Thanks
> Diana
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/0179de63/attachment-0001.html>

From knepley at gmail.com  Thu Sep 24 19:42:23 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 24 Sep 2015 19:42:23 -0500
Subject: [petsc-users] problem linking(?) a third party fortran code
 that uses petsc
In-Reply-To: <34F41684-B208-4028-B0C0-771EA817C390@astro.utoronto.ca>
References: <88D56551-9339-4AA1-B02D-A6EB41B1F690@astro.utoronto.ca>
	<CAMYG4G=T3WXoZQw44K5XsKRVuwkiAKwzLMGe25kgDmBjo97jMg@mail.gmail.com>
	<34F41684-B208-4028-B0C0-771EA817C390@astro.utoronto.ca>
Message-ID: <CAMYG4GmXcLNxVK_F5utNRYCq_wuJf+rOV9S8xC8c7at0y+-CnA@mail.gmail.com>

On Thu, Sep 24, 2015 at 7:39 PM, Diana Valencia <valencia at astro.utoronto.ca>
wrote:

> Thanks for the response.  In compiling the subroutines, one that uses
> petsc is compiled successfully with:
>
> gfortran  -fdefault-real-8 -fdefault-double-8  -w -O2    -I
> /Users/valencia/software/petsc/petsc-3.2-p7/arch-darwin-c-opt/include -c
> PETScimple_mod.f90
>

Yes, this is _compilation_, and it was successful. However, in order for
the _link_ to be successful, you
will need the PETSc library on the link line. I am guessing that there is a
make variable you must define.

I am Cc'ing one of the people at ETH since I think that place is
responsible for this irresponsible build system.

   Matt


> _______________________
> Diana Valencia
> Assistant Professor,  Physics & Astrophysics
> Department of Physical and Environmental Sciences
> University of Toronto, Scarborough
> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> Voice 416 208-2986
>
> On Sep 24, 2015, at 8:34 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Thu, Sep 24, 2015 at 7:21 PM, Diana Valencia <
> valencia at astro.utoronto.ca> wrote:
>
>> Hi,
>>
>>
>> I am trying to compile a third party software written in fortran90 that
>> uses petsc.  There is the option of compiling the code with or without
>> petsc.  Without it I have no problems, however when I use petsc, I
>> encounter this:
>>
>> After building the subroutines the error is:
>>
>
> Nowhere in this link line do I see -lpetsc. Are you sure you have this
> build system configured correctly?
>
>    Matt
>
>
>> gfortran  -fdefault-real-8 -fdefault-double-8  -w -O2    -o stagyy main.o
>> advdif.o communication.o continents.o evocore.o geometry.o init.o io.o
>> iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o
>> poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o
>> residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o
>> implicit_diffusion.o linpack.o plot.o wrtpng.o  blas123.o lapack.o
>> nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o
>> composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o
>> io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o
>> phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o
>> timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o
>> platesLea_mod.o /opt/X11/lib/libpng.dylib
>> Undefined symbols for architecture x86_64:
>>   "_kspcreate_", referenced from:
>>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>>   "_kspdestroy_", referenced from:
>>       ___petscimple_MOD_simplepetsc_destroy_everything in
>> PETScimple_mod.o
>>   "_kspgetpc_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_kspsetfromoptions_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_kspsetoperators_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_kspsettype_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_kspsolve_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_matassemblybegin_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_matassemblyend_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_matcreateseqaij_", referenced from:
>>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>>   "_matdestroy_", referenced from:
>>       ___petscimple_MOD_simplepetsc_destroy_everything in
>> PETScimple_mod.o
>>   "_matsetvalues_", referenced from:
>>       ___petscimple_MOD_simplepetsc_insert_compactrow in PETScimple_mod.o
>>   "_mpi_comm_size_", referenced from:
>>       ___petscimple_MOD_initialize in PETScimple_mod.o
>>   "_pcfactorsetmatsolverpackage_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_pcsettype_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_petscfinalize_", referenced from:
>>       ___petscimple_MOD_finalize in PETScimple_mod.o
>>   "_petscinitialize_", referenced from:
>>       ___petscimple_MOD_initialize in PETScimple_mod.o
>>   "_veccreateseqwitharray_", referenced from:
>>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>>   "_vecdestroy_", referenced from:
>>       ___petscimple_MOD_simplepetsc_destroy_everything in
>> PETScimple_mod.o
>>   "_vecplacearray_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_vecresetarray_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>> ld: symbol(s) not found for architecture x86_64
>> collect2: error: ld returned 1 exit status
>> make: *** [stagyy] Error 1
>>
>>
>>
>> My machine is OS X 10.10 and I am working with 64 bytes.  I configured
>> petsc with the following flags:
>>
>> ./configure --with-fc=gfortran --with-fortran-datatypes --with-x=0
>> --download-mpich --download-parmetis --download-superlu
>> --download-superlu_dist --download-blacs --download-scalapack
>> --download-mumps --download-umfpack --download-ml --with-debugging=no
>> --download-f-blas-lapack=yes
>>
>> Any help is greatly appreciated!!
>> Thanks
>> Diana
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/05a6b76a/attachment.html>

From valencia at astro.utoronto.ca  Thu Sep 24 19:54:37 2015
From: valencia at astro.utoronto.ca (Diana Valencia)
Date: Thu, 24 Sep 2015 20:54:37 -0400
Subject: [petsc-users] problem linking(?) a third party fortran code
	that uses petsc
In-Reply-To: <CAMYG4GmXcLNxVK_F5utNRYCq_wuJf+rOV9S8xC8c7at0y+-CnA@mail.gmail.com>
References: <88D56551-9339-4AA1-B02D-A6EB41B1F690@astro.utoronto.ca>
	<CAMYG4G=T3WXoZQw44K5XsKRVuwkiAKwzLMGe25kgDmBjo97jMg@mail.gmail.com>
	<34F41684-B208-4028-B0C0-771EA817C390@astro.utoronto.ca>
	<CAMYG4GmXcLNxVK_F5utNRYCq_wuJf+rOV9S8xC8c7at0y+-CnA@mail.gmail.com>
Message-ID: <0EA460C6-5B03-4D6B-BCD7-60F23CFA3734@astro.utoronto.ca>

Hi
Yes, I was missing a make variable (linking to the  conf/variables directory).
thank you very much!
Diana

_______________________
Diana Valencia
Assistant Professor,  Physics & Astrophysics
Department of Physical and Environmental Sciences 
University of Toronto, Scarborough
1265 Military Trail, Toronto, ON, Canada, M1C 1A4
Voice 416 208-2986

> On Sep 24, 2015, at 8:42 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Thu, Sep 24, 2015 at 7:39 PM, Diana Valencia <valencia at astro.utoronto.ca <mailto:valencia at astro.utoronto.ca>> wrote:
> Thanks for the response.  In compiling the subroutines, one that uses petsc is compiled successfully with:
> 
> gfortran  -fdefault-real-8 -fdefault-double-8  -w -O2    -I /Users/valencia/software/petsc/petsc-3.2-p7/arch-darwin-c-opt/include -c PETScimple_mod.f90
> 
> Yes, this is _compilation_, and it was successful. However, in order for the _link_ to be successful, you
> will need the PETSc library on the link line. I am guessing that there is a make variable you must define.
> 
> I am Cc'ing one of the people at ETH since I think that place is responsible for this irresponsible build system.
> 
>    Matt
>  
> _______________________
> Diana Valencia
> Assistant Professor,  Physics & Astrophysics
> Department of Physical and Environmental Sciences 
> University of Toronto, Scarborough
> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> Voice 416 208-2986 <tel:416%20208-2986>
>> On Sep 24, 2015, at 8:34 PM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>> 
>> On Thu, Sep 24, 2015 at 7:21 PM, Diana Valencia <valencia at astro.utoronto.ca <mailto:valencia at astro.utoronto.ca>> wrote:
>> Hi,
>> 
>> 
>> I am trying to compile a third party software written in fortran90 that uses petsc.  There is the option of compiling the code with or without petsc.  Without it I have no problems, however when I use petsc, I encounter this:
>> 
>> After building the subroutines the error is:
>> 
>> Nowhere in this link line do I see -lpetsc. Are you sure you have this build system configured correctly?
>> 
>>    Matt
>>  
>> gfortran  -fdefault-real-8 -fdefault-double-8  -w -O2    -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o  blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o /opt/X11/lib/libpng.dylib   
>> Undefined symbols for architecture x86_64:
>>   "_kspcreate_", referenced from:
>>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>>   "_kspdestroy_", referenced from:
>>       ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
>>   "_kspgetpc_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_kspsetfromoptions_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_kspsetoperators_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_kspsettype_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_kspsolve_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_matassemblybegin_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_matassemblyend_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_matcreateseqaij_", referenced from:
>>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>>   "_matdestroy_", referenced from:
>>       ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
>>   "_matsetvalues_", referenced from:
>>       ___petscimple_MOD_simplepetsc_insert_compactrow in PETScimple_mod.o
>>   "_mpi_comm_size_", referenced from:
>>       ___petscimple_MOD_initialize in PETScimple_mod.o
>>   "_pcfactorsetmatsolverpackage_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_pcsettype_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_petscfinalize_", referenced from:
>>       ___petscimple_MOD_finalize in PETScimple_mod.o
>>   "_petscinitialize_", referenced from:
>>       ___petscimple_MOD_initialize in PETScimple_mod.o
>>   "_veccreateseqwitharray_", referenced from:
>>       ___petscimple_MOD_simplepetsc_create_everything in PETScimple_mod.o
>>   "_vecdestroy_", referenced from:
>>       ___petscimple_MOD_simplepetsc_destroy_everything in PETScimple_mod.o
>>   "_vecplacearray_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>>   "_vecresetarray_", referenced from:
>>       ___petscimple_MOD_simplepetsc_solve in PETScimple_mod.o
>> ld: symbol(s) not found for architecture x86_64
>> collect2: error: ld returned 1 exit status
>> make: *** [stagyy] Error 1
>> 
>> 
>> 
>> My machine is OS X 10.10 and I am working with 64 bytes.  I configured petsc with the following flags:
>> 
>> ./configure --with-fc=gfortran --with-fortran-datatypes --with-x=0 --download-mpich --download-parmetis --download-superlu --download-superlu_dist --download-blacs --download-scalapack --download-mumps --download-umfpack --download-ml --with-debugging=no --download-f-blas-lapack=yes
>> 
>> Any help is greatly appreciated!!
>> Thanks
>> Diana
>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/2f3ce9f2/attachment-0001.html>

From jychang48 at gmail.com  Thu Sep 24 20:21:06 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Thu, 24 Sep 2015 19:21:06 -0600
Subject: [petsc-users] Hypre's BoomerAMG vs PETSc's GAMG?
In-Reply-To: <CAP2=TMhsZ+hn-_8r0YBH5Y5R+Ow+C+tS5SN-PC--Hbrc+aSbmw@mail.gmail.com>
References: <CAP2=TMhGpE8UkvpTuB7uuvcwRU=4Jhb+at=oyiaRgPyz7pkZ5A@mail.gmail.com>
	<87zj0k6jti.fsf@jedbrown.org>
	<CADOhEh6D5wcLCmiHqDL2S1qZ+7VgvA_bQdMgQxzGJk8-vXCLXQ@mail.gmail.com>
	<CAP2=TMhsZ+hn-_8r0YBH5Y5R+Ow+C+tS5SN-PC--Hbrc+aSbmw@mail.gmail.com>
Message-ID: <CAP2=TMiKc2Ld7=Ue43=wXgq6K2wSCZ=CX6MD59=zp=ywAe2R0g@mail.gmail.com>

Hi all,

I ran both BoomerAMG and GAMG. It seems to me there is a lot more "info" in
GAMG's log_summary than the one for Hypre's BoomerAMG. Is the latter
completely documented within PETSc's log_summary? Because I am not sure
these performance metrics make sense to me.

Attached are the -log_summary and -info outputs for the respective solvers.

Thanks,
Justin

On Mon, Sep 21, 2015 at 1:25 PM, Justin Chang <jychang48 at gmail.com> wrote:

> Okay I will just experiment around with these then. Thanks for the input
> everyone
>
> On Mon, Sep 21, 2015 at 6:21 AM, Mark Adams <mfadams at lbl.gov> wrote:
>
>> And I will just add that performance is sensitive to parameters. The
>> defaults try to be conservative and hypre's seem to be geared for 2D low
>> order discretizations.  GAMG is probably a bit more geared for 3D.  If you
>> are interested in looking this carefully you can run GAMG with '-info', and
>> grep on GAMG, and send us the results and we can verify that it running OK.
>> Also run with -log_summary and send that separately.  The two solvers
>> should be about the same speed on your problem.
>>
>>
>> On Fri, Sep 18, 2015 at 7:36 AM, Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Justin Chang <jychang48 at gmail.com> writes:
>>> > I see that there's GAMG and there's Hypre's BoomerAMG (and perhaps
>>> others
>>> > too?)
>>>
>>> ML
>>>
>>> > What exactly is the difference between these two? Do they have very
>>> > different implementations under the hood? Does one have better
>>> > scalability over another? Or more importantly, when would I choose one
>>> > over the other?
>>>
>>> BoomerAMG is classical AMG, which is a different coarsening strategy
>>> From GAMG and ML which use smoothed aggregation.  The math is different
>>> and the implementation is different.  ML and GAMG have similar math and
>>> there exist configurations that are equivalent.
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/4982f8bd/attachment-0001.html>
-------------- next part --------------
[0] PetscInitialize(): PETSc successfully started: number of processors = 1
[0] PetscInitialize(): Running on machine: Justins-MacBook-Pro.local
[0] PetscCommDuplicate(): Duplicating a communicator 4561956480 140344159520272 max tags = 2147483647
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Duplicating a communicator 4561957504 140344119264720 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140344119264720
[0] Petsc_DelComm_Outer(): User MPI_Comm 4561957504 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140344119264720
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140344119264720
[0] PetscCommDuplicate(): Duplicating a communicator 4561957504 140344119264720 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140344119264720
[0] Petsc_DelComm_Outer(): User MPI_Comm 4561957504 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140344119264720
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140344119264720
[0] PetscCommDuplicate(): Duplicating a communicator 4561957504 140344119264720 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140344119264720
[0] Petsc_DelComm_Outer(): User MPI_Comm 4561957504 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140344119264720
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140344119264720
[0] PetscCommDuplicate(): Duplicating a communicator 4561957504 140344158987552 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140344158987552
[0] Petsc_DelComm_Outer(): User MPI_Comm 4561957504 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140344158987552
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140344158987552
[0] PetscCommDuplicate(): Duplicating a communicator 4561957504 140344159036880 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140344159036880
[0] Petsc_DelComm_Outer(): User MPI_Comm 4561957504 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140344159036880
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140344159036880
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Duplicating a communicator 4561957504 140344159525648 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140344159525648
[0] Petsc_DelComm_Outer(): User MPI_Comm 4561957504 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140344159525648
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140344159525648
[0] PetscCommDuplicate(): Duplicating a communicator 4561957504 140344159526976 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140344159526976
[0] Petsc_DelComm_Outer(): User MPI_Comm 4561957504 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140344159526976
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140344159526976
[0] PetscCommDuplicate(): Duplicating a communicator 4561957504 140344159526976 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140344159526976
[0] Petsc_DelComm_Outer(): User MPI_Comm 4561957504 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140344159526976
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140344159526976
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Duplicating a communicator 4561957504 140344162264384 max tags = 2147483647
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] MatSetOption_SeqAIJ(): Option NEW_NONZERO_LOCATIONS ignored
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,181804 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 18
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10302 nodes out of 10302 rows. Not using Inode routines
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] MatSetOption_SeqAIJ(): Option NEW_NONZERO_LOCATIONS ignored
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,181804 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 18
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10302 nodes out of 10302 rows. Not using Inode routines
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,181804 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 18
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,181804 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 18
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
	Time = 100.000000
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp_GAMG(): level 0) N=10302, n data rows=1, n data cols=1, nnz/row (ave)=18, np=1
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 91706 unneeded,90098 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10302 nodes out of 10302 rows. Not using Inode routines
[0] PCGAMGFilterGraph(): 	 49.5578% nnz after filtering, with threshold 0, 17.6474 nnz ave. (N=10302)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,90098 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10302 nodes out of 10302 rows. Not using Inode routines
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,246114 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 25
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10302 nodes out of 10302 rows. Not using Inode routines
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Reallocs 0; Fill ratio: given 2 needed 1.36582.
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Use MatMatMult(A,B,MatReuse,1.36582,&C) for best performance.;
[0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,246114 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 25
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] maxIndSetAgg(): 	 removed 102 of 10302 vertices.  801 selected.
[0] PCGAMGProlongator_AGG(): New grid 801 nodes
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 801; storage space: 102 unneeded,10200 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 102)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 3342 nodes of 10302. Limit used: 5. Using Inode routines
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.944119e+00 min=7.255192e-02 PC=jacobi
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 801; storage space: 0 unneeded,50460 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 9108 nodes out of 10302 rows. Not using Inode routines
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Use MatMatMult(A,B,MatReuse,1,&C) for best performance.;
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 801; storage space: 0 unneeded,50460 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 801; storage space: 0 unneeded,50460 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 801 X 801; storage space: 0 unneeded,17709 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 33
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 801) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 801 nodes out of 801 rows. Not using Inode routines
[0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Use MatPtAP(A,P,MatReuse,1,&C) for best performance.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 801 X 801; storage space: 0 unneeded,17709 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 33
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 801) < 0.6. Do not use CompressedRow routines.
[0] PCSetUp_GAMG(): 1) N=801, n data cols=1, nnz/row (ave)=22, 1 active pes
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 801 X 801; storage space: 9106 unneeded,8603 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 17
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 801) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 801 nodes out of 801 rows. Not using Inode routines
[0] PCGAMGFilterGraph(): 	 48.5798% nnz after filtering, with threshold 0, 22.1086 nnz ave. (N=801)
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] maxIndSetAgg(): 	 removed 0 of 801 vertices.  138 selected.
[0] PCGAMGProlongator_AGG(): New grid 138 nodes
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 801 X 138; storage space: 0 unneeded,801 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 801) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 601 nodes of 801. Limit used: 5. Using Inode routines
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.910989e+00 min=5.766908e-02 PC=jacobi
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 801 X 138; storage space: 0 unneeded,6120 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 801) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 787 nodes out of 801 rows. Not using Inode routines
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Use MatMatMult(A,B,MatReuse,1,&C) for best performance.;
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 801 X 138; storage space: 0 unneeded,6120 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 801) < 0.6. Do not use CompressedRow routines.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 801 X 138; storage space: 0 unneeded,6120 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 801) < 0.6. Do not use CompressedRow routines.
[0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 138 X 138; storage space: 0 unneeded,5350 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 66
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 138) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 137 nodes out of 138 rows. Not using Inode routines
[0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Use MatPtAP(A,P,MatReuse,1,&C) for best performance.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 138 X 138; storage space: 0 unneeded,5350 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 66
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 138) < 0.6. Do not use CompressedRow routines.
[0] PCSetUp_GAMG(): 2) N=138, n data cols=1, nnz/row (ave)=38, 1 active pes
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 138 X 138; storage space: 2918 unneeded,2432 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 33
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 138) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 137 nodes out of 138 rows. Not using Inode routines
[0] PCGAMGFilterGraph(): 	 45.4579% nnz after filtering, with threshold 0, 38.7681 nnz ave. (N=138)
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] maxIndSetAgg(): 	 removed 0 of 138 vertices.  20 selected.
[0] PCGAMGProlongator_AGG(): New grid 20 nodes
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 138 X 20; storage space: 0 unneeded,138 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 138) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 79 nodes of 138. Limit used: 5. Using Inode routines
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.601213e+00 min=2.765731e-02 PC=jacobi
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 138 X 20; storage space: 0 unneeded,1074 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 138) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 125 nodes out of 138 rows. Not using Inode routines
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Use MatMatMult(A,B,MatReuse,1,&C) for best performance.;
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 138 X 20; storage space: 0 unneeded,1074 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 138) < 0.6. Do not use CompressedRow routines.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 138 X 20; storage space: 0 unneeded,1074 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 138) < 0.6. Do not use CompressedRow routines.
[0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 20 X 20; storage space: 0 unneeded,388 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 20
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 20) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10 nodes of 20. Limit used: 5. Using Inode routines
[0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Use MatPtAP(A,P,MatReuse,1,&C) for best performance.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 20 X 20; storage space: 0 unneeded,388 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 20
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 20) < 0.6. Do not use CompressedRow routines.
[0] PCSetUp_GAMG(): 3) N=20, n data cols=1, nnz/row (ave)=19, 1 active pes
[0] PCSetUp_GAMG(): 4 levels, grid complexity = 1.12897
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] PCSetUp(): Setting up PC for first time
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PCSetUp_MG(): Using outer operators to define finest grid operator 
  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Setting up PC for first time
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 5 needed 1
[0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 1 or use 
[0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,1);
[0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
[0] MatSeqAIJCheckInode_FactorLU(): Found 10 nodes of 20. Limit used: 5. Using Inode routines
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] KSPConvergedDefault(): Linear solver has converged. Residual norm 3.809939815562e-06 is less than relative tolerance 1.000000000000e-07 times initial right hand side norm 4.066392182486e+01 at iteration 50
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp_GAMG(): level 0) N=10302, n data rows=1, n data cols=1, nnz/row (ave)=18, np=1
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 94882 unneeded,86922 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10302 nodes out of 10302 rows. Not using Inode routines
[0] PCGAMGFilterGraph(): 	 47.8108% nnz after filtering, with threshold 0, 17.6474 nnz ave. (N=10302)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,86922 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10302 nodes out of 10302 rows. Not using Inode routines
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,236634 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 25
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10302 nodes out of 10302 rows. Not using Inode routines
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Reallocs 0; Fill ratio: given 2 needed 1.3612.
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Use MatMatMult(A,B,MatReuse,1.3612,&C) for best performance.;
[0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,236634 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 25
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] maxIndSetAgg(): 	 removed 502 of 10302 vertices.  773 selected.
[0] PCGAMGProlongator_AGG(): New grid 773 nodes
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 773; storage space: 502 unneeded,9800 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 502)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 3808 nodes of 10302. Limit used: 5. Using Inode routines
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.917222e+00 min=6.952647e-02 PC=jacobi
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 773; storage space: 0 unneeded,49752 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 8838 nodes out of 10302 rows. Not using Inode routines
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Use MatMatMult(A,B,MatReuse,1,&C) for best performance.;
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 773; storage space: 0 unneeded,49752 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 773; storage space: 0 unneeded,49752 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 773 X 773; storage space: 0 unneeded,17051 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 33
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 773) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 773 nodes out of 773 rows. Not using Inode routines
[0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Use MatPtAP(A,P,MatReuse,1,&C) for best performance.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 773 X 773; storage space: 0 unneeded,17051 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 33
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 773) < 0.6. Do not use CompressedRow routines.
[0] PCSetUp_GAMG(): 1) N=773, n data cols=1, nnz/row (ave)=22, 1 active pes
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 773 X 773; storage space: 8762 unneeded,8289 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 17
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 773) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 773 nodes out of 773 rows. Not using Inode routines
[0] PCGAMGFilterGraph(): 	 48.613% nnz after filtering, with threshold 0, 22.0582 nnz ave. (N=773)
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] maxIndSetAgg(): 	 removed 0 of 773 vertices.  142 selected.
[0] PCGAMGProlongator_AGG(): New grid 142 nodes
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 773 X 142; storage space: 0 unneeded,773 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 773) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 580 nodes of 773. Limit used: 5. Using Inode routines
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.935373e+00 min=5.212847e-02 PC=jacobi
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 773 X 142; storage space: 0 unneeded,5912 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 14
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 773) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 766 nodes out of 773 rows. Not using Inode routines
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Use MatMatMult(A,B,MatReuse,1,&C) for best performance.;
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 773 X 142; storage space: 0 unneeded,5912 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 14
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 773) < 0.6. Do not use CompressedRow routines.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 773 X 142; storage space: 0 unneeded,5912 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 14
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 773) < 0.6. Do not use CompressedRow routines.
[0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 142 X 142; storage space: 0 unneeded,5436 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 65
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 142) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 142 nodes out of 142 rows. Not using Inode routines
[0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Use MatPtAP(A,P,MatReuse,1,&C) for best performance.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 142 X 142; storage space: 0 unneeded,5436 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 65
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 142) < 0.6. Do not use CompressedRow routines.
[0] PCSetUp_GAMG(): 2) N=142, n data cols=1, nnz/row (ave)=38, 1 active pes
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 142 X 142; storage space: 2946 unneeded,2490 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 31
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 142) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 142 nodes out of 142 rows. Not using Inode routines
[0] PCGAMGFilterGraph(): 	 45.8057% nnz after filtering, with threshold 0, 38.2817 nnz ave. (N=142)
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] maxIndSetAgg(): 	 removed 0 of 142 vertices.  13 selected.
[0] PCGAMGProlongator_AGG(): New grid 13 nodes
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 142 X 13; storage space: 0 unneeded,142 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 142) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 86 nodes of 142. Limit used: 5. Using Inode routines
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.625439e+00 min=2.992143e-02 PC=jacobi
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 142 X 13; storage space: 0 unneeded,885 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 10
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 142) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 123 nodes out of 142 rows. Not using Inode routines
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatMatMultSymbolic_SeqAIJ_SeqAIJ(): Use MatMatMult(A,B,MatReuse,1,&C) for best performance.;
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 142 X 13; storage space: 0 unneeded,885 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 10
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 142) < 0.6. Do not use CompressedRow routines.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 142 X 13; storage space: 0 unneeded,885 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 10
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 142) < 0.6. Do not use CompressedRow routines.
[0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 13 X 13; storage space: 0 unneeded,169 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 13) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 3 nodes of 13. Limit used: 5. Using Inode routines
[0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Reallocs 0; Fill ratio: given 2 needed 1.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ_SparseAxpy(): Use MatPtAP(A,P,MatReuse,1,&C) for best performance.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 13 X 13; storage space: 0 unneeded,169 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 13) < 0.6. Do not use CompressedRow routines.
[0] PCSetUp_GAMG(): 3) N=13, n data cols=1, nnz/row (ave)=13, 1 active pes
[0] PCSetUp_GAMG(): 4 levels, grid complexity = 1.12462
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] PCSetUp(): Setting up PC for first time
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PCSetUp_MG(): Using outer operators to define finest grid operator 
  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] KSPSetNormType(): Warning: setting KSPNormType to skip computing the norm
 KSP convergence test is implicitly set to KSPConvergedSkip
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Setting up PC for first time
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561957504 140344162264384
[0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 5 needed 1
[0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 1 or use 
[0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,1);
[0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
[0] MatSeqAIJCheckInode_FactorLU(): Found 3 nodes of 13. Limit used: 5. Using Inode routines
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] KSPConvergedDefault(): Linear solver has converged. Residual norm 3.083449067853e-06 is less than relative tolerance 1.000000000000e-07 times initial right hand side norm 3.534731080542e+01 at iteration 37
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
[0] PetscCommDuplicate(): Using internal PETSc communicator 4561956480 140344159520272
	Wall-clock time: 1.018e+00 seconds
[0] PetscFinalize(): PetscFinalize() called
-------------- next part --------------
	Time = 100.000000
	Wall-clock time: 1.042e+00 seconds
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

2D_plume_ADR_ex1.py on a arch-python-macosx-10.10-x86_64 named Justins-MacBook-Pro.local with 1 processor, by justin Thu Sep 24 19:07:45 2015
Using Petsc Development GIT revision: v3.5.2-4224-gf2b0407  GIT Date: 2015-08-27 23:56:02 -0500

                         Max       Max/Min        Avg      Total 
Time (sec):           1.948e+00      1.00000   1.948e+00
Objects:              5.680e+02      1.00000   5.680e+02
Flops:                7.381e+08      1.00000   7.381e+08  7.381e+08
Flops/sec:            3.788e+08      1.00000   3.788e+08  3.788e+08
MPI Messages:         0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Message Lengths:  0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Reductions:       0.000e+00      0.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 1.1395e+00  58.5%  7.4010e+07  10.0%  0.000e+00   0.0%  0.000e+00        0.0%  0.000e+00   0.0% 
 1:       Advection: 4.3978e-01  22.6%  3.4758e+08  47.1%  0.000e+00   0.0%  0.000e+00        0.0%  0.000e+00   0.0% 
 2:       Diffusion: 3.6915e-01  18.9%  3.1646e+08  42.9%  0.000e+00   0.0%  0.000e+00        0.0%  0.000e+00   0.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

VecSet                11 1.0 3.1638e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin       11 1.0 5.2452e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin      12 1.0 1.9073e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd        12 1.0 7.2503e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DMPlexInterp           1 1.0 1.0208e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
DMPlexStratify         2 1.0 3.1607e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFSetGraph             7 1.0 4.3464e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0

--- Event Stage 1: Advection

VecMDot               60 1.0 5.3263e-04 1.0 2.47e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  4642
VecTDot              100 1.0 4.9376e-04 1.0 2.06e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  4173
VecNorm              117 1.0 3.2520e-04 1.0 1.55e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4752
VecScale              66 1.0 7.4863e-05 1.0 2.47e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3303
VecCopy              161 1.0 4.1890e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               619 1.0 2.2171e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecAXPY              106 1.0 7.0024e-04 1.0 2.11e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  3007
VecAYPX             1273 1.0 3.8307e-03 1.0 6.74e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   1  2  0  0  0  1760
VecAXPBYCZ           612 1.0 2.4939e-03 1.0 1.15e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   1  3  0  0  0  4598
VecMAXPY              66 1.0 5.5337e-04 1.0 2.92e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  5282
VecAssemblyBegin       3 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAssemblyEnd         3 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult      33 1.0 1.4234e-04 1.0 1.24e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   869
VecScatterBegin        6 1.0 3.8147e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize          66 1.0 1.7452e-04 1.0 7.42e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4251
MatMult             1028 1.0 7.6496e-02 1.0 1.48e+08 1.0 0.0e+00 0.0e+00 0.0e+00  4 20  0  0  0  17 42  0  0  0  1929
MatMultAdd           153 1.0 7.4027e-03 1.0 5.88e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   2  2  0  0  0   794
MatMultTranspose     153 1.0 7.9162e-03 1.0 5.88e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   2  2  0  0  0   743
MatSolve              51 1.0 1.5569e-04 1.0 3.86e+04 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   248
MatSOR               951 1.0 1.0649e-01 1.0 1.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00  5 18  0  0  0  24 37  0  0  0  1220
MatLUFactorSym         1 1.0 2.1935e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 4.7922e-05 1.0 2.14e+03 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0    45
MatConvert             3 1.0 1.6639e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale               9 1.0 4.6420e-04 1.0 5.25e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1131
MatResidual          153 1.0 1.0476e-02 1.0 2.09e+07 1.0 0.0e+00 0.0e+00 0.0e+00  1  3  0  0  0   2  6  0  0  0  1995
MatAssemblyBegin      24 1.0 4.0531e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd        24 1.0 2.0115e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRow          44964 1.0 3.9079e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatGetRowIJ            1 1.0 6.9141e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 3.0994e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             3 1.0 5.0902e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAXPY                3 1.0 2.3091e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatMatMult             3 1.0 4.0581e-03 1.0 4.65e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0   115
MatMatMultSym          3 1.0 2.8350e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatMatMultNum          3 1.0 1.2052e-03 1.0 4.65e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   386
MatPtAP                3 1.0 1.8055e-02 1.0 3.33e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   4  1  0  0  0   184
MatPtAPSymbolic        3 1.0 5.2621e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatPtAPNumeric         3 1.0 1.2783e-02 1.0 3.33e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   3  1  0  0  0   260
MatTrnMatMult          1 1.0 7.5231e-03 1.0 1.60e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0   212
MatTrnMatMultSym       1 1.0 5.3370e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatTrnMatMultNum       1 1.0 2.1839e-03 1.0 1.60e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   732
MatGetSymTrans         4 1.0 6.4635e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCGAMGGraph_AGG        3 1.0 1.1266e-02 1.0 4.10e+05 1.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   3  0  0  0  0    36
PCGAMGCoarse_AGG       3 1.0 8.6429e-03 1.0 1.60e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0   185
PCGAMGProl_AGG         3 1.0 1.3149e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCGAMGPOpt_AGG         3 1.0 1.5178e-02 1.0 7.78e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   3  2  0  0  0   513
GAMG: createProl       3 1.0 3.6541e-02 1.0 9.79e+06 1.0 0.0e+00 0.0e+00 0.0e+00  2  1  0  0  0   8  3  0  0  0   268
  Graph                6 1.0 1.1213e-02 1.0 4.10e+05 1.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   3  0  0  0  0    37
  MIS/Agg              3 1.0 5.4193e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
  SA: col data         3 1.0 2.5749e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
  SA: frmProl0         3 1.0 1.2050e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
  SA: smooth           3 1.0 1.5178e-02 1.0 7.78e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   3  2  0  0  0   513
GAMG: partLevel        3 1.0 1.8073e-02 1.0 3.33e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   4  1  0  0  0   184
PCSetUp                2 1.0 5.5268e-02 1.0 1.31e+07 1.0 0.0e+00 0.0e+00 0.0e+00  3  2  0  0  0  13  4  0  0  0   237
PCSetUpOnBlocks       51 1.0 1.4710e-04 1.0 2.14e+03 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0    15
PCApply               51 1.0 2.0163e-01 1.0 2.88e+08 1.0 0.0e+00 0.0e+00 0.0e+00 10 39  0  0  0  46 83  0  0  0  1428
KSPGMRESOrthog        60 1.0 1.0383e-03 1.0 4.95e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4763
KSPSetUp              12 1.0 7.9489e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 2.6766e-01 1.0 3.25e+08 1.0 0.0e+00 0.0e+00 0.0e+00 14 44  0  0  0  61 93  0  0  0  1214

--- Event Stage 2: Diffusion

VecMDot               60 1.0 6.3133e-04 1.0 2.47e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  3908
VecTDot               74 1.0 3.8075e-04 1.0 1.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4004
VecNorm              104 1.0 2.5749e-04 1.0 1.28e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4957
VecScale              66 1.0 7.4387e-05 1.0 2.47e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3317
VecCopy              122 1.0 3.3307e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               501 1.0 1.9374e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecAXPY               80 1.0 5.5432e-04 1.0 1.57e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2831
VecAYPX              948 1.0 3.0065e-03 1.0 5.00e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   1  2  0  0  0  1664
VecAXPBYCZ           456 1.0 1.7724e-03 1.0 8.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  3  0  0  0  4810
VecMAXPY              66 1.0 6.7997e-04 1.0 2.92e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  4289
VecAssemblyBegin       3 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAssemblyEnd         3 1.0 9.5367e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult      33 1.0 1.2732e-04 1.0 1.23e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   969
VecScatterBegin        6 1.0 3.2902e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize          66 1.0 1.6809e-04 1.0 7.40e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4404
MatMult              781 1.0 5.9309e-02 1.0 1.12e+08 1.0 0.0e+00 0.0e+00 0.0e+00  3 15  0  0  0  16 35  0  0  0  1882
MatMultAdd           114 1.0 5.7106e-03 1.0 4.30e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   2  1  0  0  0   753
MatMultTranspose     114 1.0 6.1491e-03 1.0 4.30e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   2  1  0  0  0   699
MatSolve              38 1.0 7.7963e-05 1.0 1.24e+04 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   158
MatSOR               717 1.0 8.3516e-02 1.0 9.77e+07 1.0 0.0e+00 0.0e+00 0.0e+00  4 13  0  0  0  23 31  0  0  0  1169
MatLUFactorSym         1 1.0 1.1921e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 5.9605e-06 1.0 9.39e+02 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   158
MatConvert             3 1.0 3.9697e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale               9 1.0 4.9543e-04 1.0 5.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1053
MatResidual          114 1.0 8.5526e-03 1.0 1.55e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   2  5  0  0  0  1815
MatAssemblyBegin      24 1.0 4.7684e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd        24 1.0 2.0373e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatGetRow          44868 1.0 3.2351e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatGetRowIJ            1 1.0 1.9073e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 2.2888e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             3 1.0 6.3801e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAXPY                3 1.0 2.1653e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatMatMult             3 1.0 3.7990e-03 1.0 4.53e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0   119
MatMatMultSym          3 1.0 2.4741e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatMatMultNum          3 1.0 1.3139e-03 1.0 4.53e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   345
MatPtAP                3 1.0 1.8103e-02 1.0 3.24e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   5  1  0  0  0   179
MatPtAPSymbolic        3 1.0 5.9602e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0     0
MatPtAPNumeric         3 1.0 1.2134e-02 1.0 3.24e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   3  1  0  0  0   267
MatTrnMatMult          1 1.0 1.0095e-02 1.0 1.54e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   3  0  0  0  0   152
MatTrnMatMultSym       1 1.0 7.6640e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0     0
MatTrnMatMultNum       1 1.0 2.4290e-03 1.0 1.54e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0   632
MatGetSymTrans         4 1.0 6.4111e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCGAMGGraph_AGG        3 1.0 8.3809e-03 1.0 4.09e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0    49
PCGAMGCoarse_AGG       3 1.0 1.1429e-02 1.0 1.54e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   3  0  0  0  0   134
PCGAMGProl_AGG         3 1.0 1.4434e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCGAMGPOpt_AGG         3 1.0 1.2587e-02 1.0 7.75e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   3  2  0  0  0   616
GAMG: createProl       3 1.0 3.3962e-02 1.0 9.69e+06 1.0 0.0e+00 0.0e+00 0.0e+00  2  1  0  0  0   9  3  0  0  0   285
  Graph                6 1.0 8.3411e-03 1.0 4.09e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0    49
  MIS/Agg              3 1.0 6.6781e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
  SA: col data         3 1.0 2.9087e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
  SA: frmProl0         3 1.0 1.3227e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
  SA: smooth           3 1.0 1.2586e-02 1.0 7.75e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   3  2  0  0  0   616
GAMG: partLevel        3 1.0 1.8109e-02 1.0 3.24e+06 1.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   5  1  0  0  0   179
PCSetUp                2 1.0 5.2608e-02 1.0 1.29e+07 1.0 0.0e+00 0.0e+00 0.0e+00  3  2  0  0  0  14  4  0  0  0   246
PCSetUpOnBlocks       38 1.0 7.8678e-05 1.0 9.39e+02 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0    12
PCApply               38 1.0 1.5786e-01 1.0 2.17e+08 1.0 0.0e+00 0.0e+00 0.0e+00  8 29  0  0  0  43 68  0  0  0  1373
KSPGMRESOrthog        60 1.0 1.2169e-03 1.0 4.94e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  2  0  0  0  4056
KSPSetUp              12 1.0 6.5565e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 2.1840e-01 1.0 2.47e+08 1.0 0.0e+00 0.0e+00 0.0e+00 11 34  0  0  0  59 78  0  0  0  1132
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

           Container     6              3         1704     0
              Viewer     1              0            0     0
           Index Set    34             32       351112     0
   IS L to G Mapping    13              0            0     0
             Section    21              3         1992     0
              Vector    26            162      4803960     0
      Vector Scatter     9             15         9840     0
              Matrix    10             14       186192     0
      Preconditioner     4             14        14088     0
       Krylov Solver     4             20       198208     0
                 Tao     2              2         3920     0
    Distributed Mesh     8              1         4904     0
    GraphPartitioner     2              1          604     0
Star Forest Bipartite Graph    19              6         4800     0
     Discrete System     8              1          848     0
       TaoLineSearch     2              2         1760     0

--- Event Stage 1: Advection

           Index Set    16             13        10168     0
              Vector   141             67      2080952     0
      Vector Scatter     4              0            0     0
              Matrix    18             11      5703204     0
      Matrix Coarsen     3              3         1884     0
      Preconditioner    11              6         5544     0
       Krylov Solver    11              3        90576     0

--- Event Stage 2: Diffusion

           Index Set    12              9         7008     0
              Vector   138             67      2076672     0
      Vector Scatter     2              0            0     0
              Matrix    18             11      5688036     0
      Matrix Coarsen     3              3         1884     0
      Preconditioner    11              6         5544     0
       Krylov Solver    11              3        90576     0
========================================================================================================================
Average time to get PetscTime(): 0
#PETSc Option Table entries:
-A_ksp_rtol 1e-7
-A_ksp_type cg
-A_pc_type gamg
-D_ksp_rtol 1e-7
-D_ksp_type cg
-D_pc_type gamg
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --prefix=/Users/justin/Software/firedrake/lib/python2.7/site-packages/petsc PETSC_ARCH=arch-python-macosx-10.10-x86_64 --with-shared-libraries=1 --with-debugging=0 --with-c2html=0 --with-cc=/usr/local/bin/mpicc --with-cxx=/usr/local/bin/mpicxx --with-fc=/usr/local/bin/mpif90 --download-ctetgen --download-triangle --download-chaco --download-metis --download-parmetis --download-scalapack --download-hypre=1 --download-mumps --download-netcdf --download-hdf5 --download-exodusii
-----------------------------------------
Libraries compiled on Fri Sep  4 01:55:38 2015 on Justins-MacBook-Pro.local 
Machine characteristics: Darwin-14.5.0-x86_64-i386-64bit
Using PETSc directory: /private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build
Using PETSc arch: arch-python-macosx-10.10-x86_64
-----------------------------------------

Using C compiler: /usr/local/bin/mpicc  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O3  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /usr/local/bin/mpif90  -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/arch-python-macosx-10.10-x86_64/include -I/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/include -I/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/include -I/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/arch-python-macosx-10.10-x86_64/include -I/Users/justin/Software/firedrake/lib/python2.7/site-packages/petsc/include -I/usr/local/Cellar/open-mpi/1.10.0/include
-----------------------------------------

Using C linker: /usr/local/bin/mpicc
Using Fortran linker: /usr/local/bin/mpif90
Using libraries: -Wl,-rpath,/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/arch-python-macosx-10.10-x86_64/lib -L/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/arch-python-macosx-10.10-x86_64/lib -lpetsc -Wl,-rpath,/Users/justin/Software/firedrake/lib/python2.7/site-packages/petsc/lib -L/Users/justin/Software/firedrake/lib/python2.7/site-packages/petsc/lib -lHYPRE -Wl,-rpath,/usr/local/opt/libevent/lib -L/usr/local/opt/libevent/lib -Wl,-rpath,/usr/local/Cellar/open-mpi/1.10.0/lib -L/usr/local/Cellar/open-mpi/1.10.0/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/6.1.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/6.1.0/lib/darwin -lclang_rt.osx -lmpi_cxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.1.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.1.0/lib/darwin -lclang_rt.osx -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -llapack -lblas -lparmetis -lmetis -ltriangle -lchaco -lexoIIv2for -lexodus -lnetcdf -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lctetgen -lssl -lcrypto -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -Wl,-rpath,/usr/local/Cellar/gcc/5.2.0/lib/gcc/5/gcc/x86_64-apple-darwin14.4.0/5.2.0 -L/usr/local/Cellar/gcc/5.2.0/lib/gcc/5/gcc/x86_64-apple-darwin14.4.0/5.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/5.2.0/lib/gcc/5 -L/usr/local/Cellar/gcc/5.2.0/lib/gcc/5 -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpi_cxx -lc++ -lclang_rt.osx -Wl,-rpath,/usr/local/opt/libevent/lib -L/usr/local/opt/libevent/lib -Wl,-rpath,/usr/local/Cellar/open-mpi/1.10.0/lib -L/usr/local/Cellar/open-mpi/1.10.0/lib -ldl -lmpi -lSystem -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.1.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.1.0/lib/darwin -lclang_rt.osx -ldl 
-----------------------------------------
-------------- next part --------------
[0] PetscInitialize(): PETSc successfully started: number of processors = 1
[0] PetscInitialize(): Running on machine: Justins-MacBook-Pro.local
[0] PetscCommDuplicate(): Duplicating a communicator 4429799040 140522300059408 max tags = 2147483647
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Duplicating a communicator 4429800064 140522300073648 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140522300073648
[0] Petsc_DelComm_Outer(): User MPI_Comm 4429800064 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140522300073648
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140522300073648
[0] PetscCommDuplicate(): Duplicating a communicator 4429800064 140522300073648 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140522300073648
[0] Petsc_DelComm_Outer(): User MPI_Comm 4429800064 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140522300073648
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140522300073648
[0] PetscCommDuplicate(): Duplicating a communicator 4429800064 140522300073648 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140522300073648
[0] Petsc_DelComm_Outer(): User MPI_Comm 4429800064 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140522300073648
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140522300073648
[0] PetscCommDuplicate(): Duplicating a communicator 4429800064 140522300072848 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140522300072848
[0] Petsc_DelComm_Outer(): User MPI_Comm 4429800064 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140522300072848
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140522300072848
[0] PetscCommDuplicate(): Duplicating a communicator 4429800064 140522300128496 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140522300128496
[0] Petsc_DelComm_Outer(): User MPI_Comm 4429800064 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140522300128496
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140522300128496
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Duplicating a communicator 4429800064 140522299392960 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140522299392960
[0] Petsc_DelComm_Outer(): User MPI_Comm 4429800064 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140522299392960
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140522299392960
[0] PetscCommDuplicate(): Duplicating a communicator 4429800064 140522299393728 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140522299393728
[0] Petsc_DelComm_Outer(): User MPI_Comm 4429800064 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140522299393728
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140522299393728
[0] PetscCommDuplicate(): Duplicating a communicator 4429800064 140522299393728 max tags = 2147483647
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm 140522299393728
[0] Petsc_DelComm_Outer(): User MPI_Comm 4429800064 is being freed after removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 140522299393728
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm 140522299393728
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Duplicating a communicator 4429800064 140522300427040 max tags = 2147483647
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429800064 140522300427040
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429800064 140522300427040
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429800064 140522300427040
[0] MatSetOption_SeqAIJ(): Option NEW_NONZERO_LOCATIONS ignored
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,181804 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 18
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10302 nodes out of 10302 rows. Not using Inode routines
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429800064 140522300427040
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429800064 140522300427040
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429800064 140522300427040
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429800064 140522300427040
[0] MatSetOption_SeqAIJ(): Option NEW_NONZERO_LOCATIONS ignored
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,181804 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 18
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatSeqAIJCheckInode(): Found 10302 nodes out of 10302 rows. Not using Inode routines
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,181804 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 18
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 10302 X 10302; storage space: 0 unneeded,181804 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 18
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 10302) < 0.6. Do not use CompressedRow routines.
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
	Time = 100.000000
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] KSPConvergedDefault(): Linear solver has converged. Residual norm 3.965904976486e-06 is less than relative tolerance 1.000000000000e-07 times initial right hand side norm 5.489878564360e+01 at iteration 48
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] VecScatterCreate(): Special case: sequential vector stride to stride
[0] PCSetUp(): Setting up PC for first time
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
[0] KSPConvergedDefault(): Linear solver has converged. Residual norm 3.460837870404e-06 is less than relative tolerance 1.000000000000e-07 times initial right hand side norm 4.917749887197e+01 at iteration 19
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
[0] PetscCommDuplicate(): Using internal PETSc communicator 4429799040 140522300059408
	Wall-clock time: 9.638e-01 seconds
[0] PetscFinalize(): PetscFinalize() called
-------------- next part --------------
	Time = 100.000000
	Wall-clock time: 9.561e-01 seconds
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

2D_plume_ADR_ex1.py on a arch-python-macosx-10.10-x86_64 named Justins-MacBook-Pro.local with 1 processor, by justin Thu Sep 24 19:08:10 2015
Using Petsc Development GIT revision: v3.5.2-4224-gf2b0407  GIT Date: 2015-08-27 23:56:02 -0500

                         Max       Max/Min        Avg      Total 
Time (sec):           1.763e+00      1.00000   1.763e+00
Objects:              2.060e+02      1.00000   2.060e+02
Flops:                1.978e+08      1.00000   1.978e+08  1.978e+08
Flops/sec:            1.122e+08      1.00000   1.122e+08  1.122e+08
MPI Messages:         0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Message Lengths:  0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Reductions:       0.000e+00      0.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 1.0402e+00  59.0%  7.4010e+07  37.4%  0.000e+00   0.0%  0.000e+00        0.0%  0.000e+00   0.0% 
 1:       Advection: 4.4468e-01  25.2%  4.5603e+07  23.0%  0.000e+00   0.0%  0.000e+00        0.0%  0.000e+00   0.0% 
 2:       Diffusion: 2.7806e-01  15.8%  7.8232e+07  39.5%  0.000e+00   0.0%  0.000e+00        0.0%  0.000e+00   0.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

VecSet                11 1.0 3.0589e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin       11 1.0 4.8637e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin      12 1.0 1.9073e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd        12 1.0 7.7915e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DMPlexInterp           1 1.0 8.5580e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
DMPlexStratify         2 1.0 2.8751e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFSetGraph             7 1.0 4.8900e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0

--- Event Stage 1: Advection

VecTDot               96 1.0 4.2796e-04 1.0 1.98e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  4  0  0  0  4622
VecNorm               49 1.0 2.4843e-04 1.0 1.01e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  2  0  0  0  4064
VecCopy                2 1.0 1.9073e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                57 1.0 3.8910e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY               96 1.0 7.1192e-04 1.0 1.98e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  4  0  0  0  2778
VecAYPX               47 1.0 3.5429e-04 1.0 9.68e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  2  0  0  0  2733
VecScatterBegin        6 1.0 3.0994e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMult               48 1.0 9.8245e-03 1.0 1.70e+07 1.0 0.0e+00 0.0e+00 0.0e+00  1  9  0  0  0   2 37  0  0  0  1726
MatConvert             1 1.0 1.3752e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 9.5367e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCSetUp                1 1.0 4.5080e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0  10  0  0  0  0     0
PCApply               49 1.0 2.2821e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 13  0  0  0  0  51  0  0  0  0     0
KSPSetUp               1 1.0 1.2302e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 2.8528e-01 1.0 2.29e+07 1.0 0.0e+00 0.0e+00 0.0e+00 16 12  0  0  0  64 50  0  0  0    80

--- Event Stage 2: Diffusion

VecTDot               38 1.0 1.5736e-04 1.0 7.83e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  4975
VecNorm               20 1.0 9.9897e-05 1.0 4.12e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  4125
VecCopy                2 1.0 2.0981e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                27 1.0 2.6774e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY               38 1.0 2.4676e-04 1.0 7.83e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  3173
VecAYPX               18 1.0 1.2612e-04 1.0 3.71e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2941
VecScatterBegin        6 1.0 3.0756e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMult               19 1.0 3.4723e-03 1.0 6.71e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  3  0  0  0   1  9  0  0  0  1933
MatConvert             1 1.0 1.6348e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatGetRowIJ            1 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCSetUp                1 1.0 4.3261e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0  16  0  0  0  0     0
PCApply               20 1.0 8.4856e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  5  0  0  0  0  31  0  0  0  0     0
KSPSetUp               1 1.0 1.2994e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 1.3248e-01 1.0 9.06e+06 1.0 0.0e+00 0.0e+00 0.0e+00  8  5  0  0  0  48 12  0  0  0    68
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

           Container     6              3         1704     0
              Viewer     1              0            0     0
           Index Set    34             26       346192     0
   IS L to G Mapping    13              0            0     0
             Section    21              3         1992     0
              Vector    26             32      1161512     0
      Vector Scatter     9             15         9840     0
              Matrix    10              0            0     0
      Preconditioner     4              4         4384     0
       Krylov Solver     4              4         4784     0
                 Tao     2              2         3920     0
    Distributed Mesh     8              1         4904     0
    GraphPartitioner     2              1          604     0
Star Forest Bipartite Graph    19              6         4800     0
     Discrete System     8              1          848     0
       TaoLineSearch     2              2         1760     0

--- Event Stage 1: Advection

           Index Set     8              8         6208     0
              Vector    11              2       167888     0
      Vector Scatter     4              0            0     0

--- Event Stage 2: Diffusion

           Index Set     4              4         3104     0
              Vector     8              2       167888     0
      Vector Scatter     2              0            0     0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
#PETSc Option Table entries:
-A_ksp_rtol 1e-7
-A_ksp_type cg
-A_pc_hypre_type boomeramg
-A_pc_type hypre
-D_ksp_rtol 1e-7
-D_ksp_type cg
-D_pc_hypre_type boomeramg
-D_pc_type hypre
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --prefix=/Users/justin/Software/firedrake/lib/python2.7/site-packages/petsc PETSC_ARCH=arch-python-macosx-10.10-x86_64 --with-shared-libraries=1 --with-debugging=0 --with-c2html=0 --with-cc=/usr/local/bin/mpicc --with-cxx=/usr/local/bin/mpicxx --with-fc=/usr/local/bin/mpif90 --download-ctetgen --download-triangle --download-chaco --download-metis --download-parmetis --download-scalapack --download-hypre=1 --download-mumps --download-netcdf --download-hdf5 --download-exodusii
-----------------------------------------
Libraries compiled on Fri Sep  4 01:55:38 2015 on Justins-MacBook-Pro.local 
Machine characteristics: Darwin-14.5.0-x86_64-i386-64bit
Using PETSc directory: /private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build
Using PETSc arch: arch-python-macosx-10.10-x86_64
-----------------------------------------

Using C compiler: /usr/local/bin/mpicc  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O3  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /usr/local/bin/mpif90  -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/arch-python-macosx-10.10-x86_64/include -I/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/include -I/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/include -I/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/arch-python-macosx-10.10-x86_64/include -I/Users/justin/Software/firedrake/lib/python2.7/site-packages/petsc/include -I/usr/local/Cellar/open-mpi/1.10.0/include
-----------------------------------------

Using C linker: /usr/local/bin/mpicc
Using Fortran linker: /usr/local/bin/mpif90
Using libraries: -Wl,-rpath,/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/arch-python-macosx-10.10-x86_64/lib -L/private/var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2m0000gn/T/pip-I00cfW-build/arch-python-macosx-10.10-x86_64/lib -lpetsc -Wl,-rpath,/Users/justin/Software/firedrake/lib/python2.7/site-packages/petsc/lib -L/Users/justin/Software/firedrake/lib/python2.7/site-packages/petsc/lib -lHYPRE -Wl,-rpath,/usr/local/opt/libevent/lib -L/usr/local/opt/libevent/lib -Wl,-rpath,/usr/local/Cellar/open-mpi/1.10.0/lib -L/usr/local/Cellar/open-mpi/1.10.0/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/6.1.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/6.1.0/lib/darwin -lclang_rt.osx -lmpi_cxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.1.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.1.0/lib/darwin -lclang_rt.osx -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -llapack -lblas -lparmetis -lmetis -ltriangle -lchaco -lexoIIv2for -lexodus -lnetcdf -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lctetgen -lssl -lcrypto -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -Wl,-rpath,/usr/local/Cellar/gcc/5.2.0/lib/gcc/5/gcc/x86_64-apple-darwin14.4.0/5.2.0 -L/usr/local/Cellar/gcc/5.2.0/lib/gcc/5/gcc/x86_64-apple-darwin14.4.0/5.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/5.2.0/lib/gcc/5 -L/usr/local/Cellar/gcc/5.2.0/lib/gcc/5 -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpi_cxx -lc++ -lclang_rt.osx -Wl,-rpath,/usr/local/opt/libevent/lib -L/usr/local/opt/libevent/lib -Wl,-rpath,/usr/local/Cellar/open-mpi/1.10.0/lib -L/usr/local/Cellar/open-mpi/1.10.0/lib -ldl -lmpi -lSystem -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.1.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.1.0/lib/darwin -lclang_rt.osx -ldl 
-----------------------------------------

From bsmith at mcs.anl.gov  Thu Sep 24 20:28:09 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 24 Sep 2015 20:28:09 -0500
Subject: [petsc-users] Hypre's BoomerAMG vs PETSc's GAMG?
In-Reply-To: <CAP2=TMiKc2Ld7=Ue43=wXgq6K2wSCZ=CX6MD59=zp=ywAe2R0g@mail.gmail.com>
References: <CAP2=TMhGpE8UkvpTuB7uuvcwRU=4Jhb+at=oyiaRgPyz7pkZ5A@mail.gmail.com>
	<87zj0k6jti.fsf@jedbrown.org>
	<CADOhEh6D5wcLCmiHqDL2S1qZ+7VgvA_bQdMgQxzGJk8-vXCLXQ@mail.gmail.com>
	<CAP2=TMhsZ+hn-_8r0YBH5Y5R+Ow+C+tS5SN-PC--Hbrc+aSbmw@mail.gmail.com>
	<CAP2=TMiKc2Ld7=Ue43=wXgq6K2wSCZ=CX6MD59=zp=ywAe2R0g@mail.gmail.com>
Message-ID: <AD0EA7DA-0A07-4FCA-A7C3-2CCBC71AB87E@mcs.anl.gov>


  Justin,

   Since the PETSc logging cannot "see" into the hypre run it doesn't have any information about the times for constructing the AMG levels, so yes is less informative than with GAMG.

   You can get some information about the decisions hypre BoomerAMG made with the option -pc_hypre_boomeramg_print_statistics

  Barry

> On Sep 24, 2015, at 8:21 PM, Justin Chang <jychang48 at gmail.com> wrote:
> 
> Hi all,
> 
> I ran both BoomerAMG and GAMG. It seems to me there is a lot more "info" in GAMG's log_summary than the one for Hypre's BoomerAMG. Is the latter completely documented within PETSc's log_summary? Because I am not sure these performance metrics make sense to me.
> 
> Attached are the -log_summary and -info outputs for the respective solvers.
> 
> Thanks,
> Justin
> 
> On Mon, Sep 21, 2015 at 1:25 PM, Justin Chang <jychang48 at gmail.com> wrote:
> Okay I will just experiment around with these then. Thanks for the input everyone
> 
> On Mon, Sep 21, 2015 at 6:21 AM, Mark Adams <mfadams at lbl.gov> wrote:
> And I will just add that performance is sensitive to parameters. The defaults try to be conservative and hypre's seem to be geared for 2D low order discretizations.  GAMG is probably a bit more geared for 3D.  If you are interested in looking this carefully you can run GAMG with '-info', and grep on GAMG, and send us the results and we can verify that it running OK. Also run with -log_summary and send that separately.  The two solvers should be about the same speed on your problem.
> 
> 
> On Fri, Sep 18, 2015 at 7:36 AM, Jed Brown <jed at jedbrown.org> wrote:
> Justin Chang <jychang48 at gmail.com> writes:
> > I see that there's GAMG and there's Hypre's BoomerAMG (and perhaps others
> > too?)
> 
> ML
> 
> > What exactly is the difference between these two? Do they have very
> > different implementations under the hood? Does one have better
> > scalability over another? Or more importantly, when would I choose one
> > over the other?
> 
> BoomerAMG is classical AMG, which is a different coarsening strategy
> From GAMG and ML which use smoothed aggregation.  The math is different
> and the implementation is different.  ML and GAMG have similar math and
> there exist configurations that are equivalent.
> 
> 
> 
> <GAMG_info.txt><GAMG_logsummary.txt><HYPRE_info.txt><HYPRE_logsummary.txt>


From gideon.simpson at gmail.com  Thu Sep 24 21:24:49 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Thu, 24 Sep 2015 22:24:49 -0400
Subject: [petsc-users] petscprintf and __float128
Message-ID: <02DA5EBE-B7F6-4A76-ADD4-CC485BA6B4A2@gmail.com>

I saw that Barry posted something about this in the dev listserv last December, but there wasn?t any follow up.  Right now, on an OS X based machine with gcc compilers, if I do a PetscPrintf with a __float128, with %g, it outputs 0.  Is there a way to get around this, other than using quadmath_snprintf?


-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/7a7d2457/attachment.html>

From knepley at gmail.com  Thu Sep 24 21:35:51 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 24 Sep 2015 21:35:51 -0500
Subject: [petsc-users] petscprintf and __float128
In-Reply-To: <02DA5EBE-B7F6-4A76-ADD4-CC485BA6B4A2@gmail.com>
References: <02DA5EBE-B7F6-4A76-ADD4-CC485BA6B4A2@gmail.com>
Message-ID: <CAMYG4Gm==m+rAaB64VrOjFnfP63cahLzRpu_ajwKzmG2cuoVKg@mail.gmail.com>

On Thu, Sep 24, 2015 at 9:24 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> I saw that Barry posted something about this in the dev listserv last
> December, but there wasn?t any follow up.  Right now, on an OS X based
> machine with gcc compilers, if I do a PetscPrintf with a __float128, with
> %g, it outputs 0.  Is there a way to get around this, other than using
> quadmath_snprintf?
>

Just cast to double.

  Matt


> -gideon
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150924/9a2d07c0/attachment.html>

From bsmith at mcs.anl.gov  Thu Sep 24 21:36:11 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 24 Sep 2015 21:36:11 -0500
Subject: [petsc-users] petscprintf and __float128
In-Reply-To: <02DA5EBE-B7F6-4A76-ADD4-CC485BA6B4A2@gmail.com>
References: <02DA5EBE-B7F6-4A76-ADD4-CC485BA6B4A2@gmail.com>
Message-ID: <B562E208-5FE5-49BF-B7D4-288009ED0E91@mcs.anl.gov>


> On Sep 24, 2015, at 9:24 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> I saw that Barry posted something about this in the dev listserv last December, but there wasn?t any follow up.  Right now, on an OS X based machine with gcc compilers, if I do a PetscPrintf with a __float128, with %g, it outputs 0.  Is there a way to get around this, other than using quadmath_snprintf?

   There are only two ways: caste with a (double)   or use the quadmath_snprintf. When printing the residual norms it is fine to just cast to doubles to look at convergence you don't need to do anything beyond the usual -snes_monitor -ksp_monitor etc (warning only PETSc LU solvers support __float128 so you can't use Superlu or MUMPs for the direct solvers). 

  I found quadmath_snprintf cumbersome to use in PETSc and not supported on all systems so didn't try to use it in PETSc.

  Barry

> 
> 
> -gideon
> 


From timothee.nicolas at gmail.com  Fri Sep 25 00:24:27 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Fri, 25 Sep 2015 14:24:27 +0900
Subject: [petsc-users] Direct inversion methods in parallel
Message-ID: <CAGi1ndTrqxGyW-J7c3zxvY=VDDHcvUMWCQwUQxwgq2ZnD9+5Pg@mail.gmail.com>

Hi all, from the manual, I get that the options

-ksp_type preonly -pc_type lu

to solve a problem by direct LU inversion are available only for sequential
matrices. Should I conclude that there is no method to try a direct
inversion of a big problem in parallel ?

I plan to use the direct inversion only as a check that my approximation to
the inverse problem is OK, because so far my algorithm which should work is
not working at all and I need to debug what is going on. Namely I use an
approximation to the linear problem using an approximate Schur complement,
and I want to know if my approximation is false or if from the start my
matrices are false.

I have tried a direct inversion on one process with the above options for a
quite small problem (12x12x40 with 8 dof), but it did not work, I suppose
because of memory limitation (output with log_summary at the end attached
just in case).

Best

Timothee NICOLAS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150925/a2cd0c29/attachment-0001.html>
-------------- next part --------------
 
 This is an implicit MHD code based on the MIPS code 
 
 Setting all options 
 
 Start: Reading HINT2 equilibrium file 
 
 DATA: lr,lz,lphi=          12          12          40
 lsymmetry=           1
 r_min,r_max=   2.7000000000000002        4.5999999999999996     
 z_min,z_max= -0.94999999999999996       0.94999999999999996     
 phi_min,phi_max=   0.0000000000000000        6.2831853071795862     
 dr,dz,dphi=  0.17272727272727267       0.17272727272727273       0.15707963267948966     
 pmax=   4.0818670444571345E-003
 bmax=   2.9663632417550203     
 
 End: Reading HINT2 equilibrium file 
 
 Creating nonlinear solver, getting geometrical info, and setting vectors 
 
 Second order centered finite differences are used  
 Allocating some arrays 
 Masks definition 
 Major radius, angles, and vacuum definition 
 Initializing PETSc Vecs with equilibrium values 
 Set the initial force local vectors (used to enforce the equilibrium) 
 Writing the unperturbed fields 
 Add a coherent perturbation to the velocity 
 Poloidal mode number of the perturbation: m =  1 
 Toroidal mode number of the perturbation: n =  1 
 Writing the perturbed fields 
 Creating the matrices. This takes some time 
 Creating the matrices took 1.7204E+00 
 Entering the main MHD Loop 
 
 Iteration number = 1 
 Time (tau_A) = 9.99999978E-03 
 CPU time used for Building TotLinmat: 7.7378E-01 
 Total CPU time since PetscInitialize: 1.7185E+02 
 CPU time used for SNESSolve: 1.7011E+02 
 Number of linear iterations :  0 
 Number of function evaluations :  1 
 Kinetic Energy =  8.863760E-17 
 Magnetic Energy =  0.000000E+00 
 
 
 Iteration number = 2 
 Time (tau_A) = 1.99999998E-02 
 CPU time used for Building TotLinmat: 7.3685E-01 
 Total CPU time since PetscInitialize: 3.3283E+02 
 CPU time used for SNESSolve: 1.6099E+02 
 Number of linear iterations :  0 
 Number of function evaluations :  1 
 Kinetic Energy =  8.863760E-17 
 Magnetic Energy =  0.000000E+00 
 
 
 Exiting the main MHD Loop 
 
 Deallocating remaining arrays 
 
 Destroying remaining Petsc elements 
 
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./mips_implicit on a arch-darwin-c-debug named iMac27Nicolas.nifs.ac.jp with 1 processor, by timotheenicolas Fri Sep 25 13:57:08 2015
Using Petsc Release Version 3.6.1, Jul, 22, 2015 

                         Max       Max/Min        Avg      Total 
Time (sec):           3.331e+02      1.00000   3.331e+02
Objects:              7.400e+01      1.00000   7.400e+01
Flops:                3.361e+11      1.00000   3.361e+11  3.361e+11
Flops/sec:            1.009e+09      1.00000   1.009e+09  1.009e+09
Memory:               2.362e+09      1.00000              2.362e+09
MPI Messages:         0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Message Lengths:  0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Reductions:       0.000e+00      0.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 3.3309e+02 100.0%  3.3613e+11 100.0%  0.000e+00   0.0%  0.000e+00        0.0%  0.000e+00   0.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------


      ##########################################################
      #                                                        #
      #                          WARNING!!!                    #
      #                                                        #
      #   This code was compiled with a debugging option,      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

SNESSolve              2 1.0 3.2958e+02 1.0 3.36e+11 1.0 0.0e+00 0.0e+00 0.0e+00 99100  0  0  0  99100  0  0  0  1020
SNESFunctionEval       2 1.0 3.2958e+02 1.0 3.36e+11 1.0 0.0e+00 0.0e+00 0.0e+00 99100  0  0  0  99100  0  0  0  1020
VecView                5 1.0 1.6438e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNorm                6 1.0 3.7619e-04 1.0 5.53e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1470
VecCopy               14 1.0 5.7548e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                24 1.0 2.5896e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                7 1.0 9.8065e-04 1.0 6.45e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   658
VecWAXPY               3 1.0 4.4243e-04 1.0 1.38e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   312
VecAssemblyBegin       5 1.0 1.7768e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAssemblyEnd         5 1.0 1.6867e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult       4 1.0 5.7678e-04 1.0 1.84e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   320
VecScatterBegin       10 1.0 1.2419e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatSolve               2 1.0 5.8557e-01 1.0 6.31e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1077
MatLUFactorSym         1 1.0 7.7204e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   2  0  0  0  0     0
MatLUFactorNum         2 1.0 3.2121e+02 1.0 3.35e+11 1.0 0.0e+00 0.0e+00 0.0e+00 96100  0  0  0  96100  0  0  0  1044
MatAssemblyBegin       3 1.0 1.1565e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd         3 1.0 3.6987e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 1.2978e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 5.1436e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp               2 1.0 1.4995e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               2 1.0 3.2957e+02 1.0 3.36e+11 1.0 0.0e+00 0.0e+00 0.0e+00 99100  0  0  0  99100  0  0  0  1020
PCSetUp                2 1.0 3.2898e+02 1.0 3.35e+11 1.0 0.0e+00 0.0e+00 0.0e+00 99100  0  0  0  99100  0  0  0  1020
PCApply                2 1.0 5.8560e-01 1.0 6.31e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1077
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

                SNES     1              1         1324     0
      SNESLineSearch     1              1          856     0
              DMSNES     2              2         1312     0
              Vector    27             27      8261968     0
      Vector Scatter     3              3         1944     0
             MatMFFD     1              1          752     0
              Matrix     3              3   1999326700     0
    Distributed Mesh     3              3        14392     0
Star Forest Bipartite Graph     6              6         4928     0
     Discrete System     3              3         2520     0
           Index Set    11             11       519936     0
   IS L to G Mapping     2              2        51920     0
       Krylov Solver     2              2         2272     0
     DMKSP interface     1              1          640     0
      Preconditioner     2              2         1968     0
              Viewer     6              5         3800     0
========================================================================================================================
Average time to get PetscTime(): 2.61003e-08
#PETSc Option Table entries:
-cimplicit 1
-dt 1e-2
-fixed_dt
-ksp_monitor
-ksp_type preonly
-log_summary
-nts 2
-pc_type lu
-snes_mf
-snes_monitor
-total_matrix
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich
-----------------------------------------
Libraries compiled on Thu Sep 24 09:58:51 2015 on iMac27Nicolas.nifs.ac.jp 
Machine characteristics: Darwin-14.5.0-x86_64-i386-64bit
Using PETSc directory: /Users/timotheenicolas/PETSC/petsc-3.6.1
Using PETSc arch: arch-darwin-c-debug
-----------------------------------------

Using C compiler: /Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/bin/mpicc  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/bin/mpif90  -fPIC  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0  ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/include -I/Users/timotheenicolas/PETSC/petsc-3.6.1/include -I/Users/timotheenicolas/PETSC/petsc-3.6.1/include -I/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/include -I/opt/X11/include
-----------------------------------------

Using C linker: /Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/bin/mpicc
Using Fortran linker: /Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/bin/mpif90
Using libraries: -Wl,-rpath,/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -L/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -lpetsc -Wl,-rpath,/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -L/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -lflapack -lfblas -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib -lX11 -lssl -lcrypto -Wl,-rpath,/Library/Developer/CommandLineTools/usr/lib/clang/7.0.0/lib/darwin -L/Library/Developer/CommandLineTools/usr/lib/clang/7.0.0/lib/darwin -lmpifort -lgfortran -Wl,-rpath,/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0 -L/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0 -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpicxx -lc++ -Wl,-rpath,/Library/Developer/CommandLineTools/usr/bin/../lib/clang/7.0.0/lib/darwin -L/Library/Developer/CommandLineTools/usr/bin/../lib/clang/7.0.0/lib/darwin -lclang_rt.osx -Wl,-rpath,/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -L/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -ldl -lmpi -lpmpi -lSystem -Wl,-rpath,/Library/Developer/CommandLineTools/usr/bin/../lib/clang/7.0.0/lib/darwin -L/Library/Developer/CommandLineTools/usr/bin/../lib/clang/7.0.0/lib/darwin -lclang_rt.osx -ldl 
-----------------------------------------

WARNING! There are options you set that were not used!
WARNING! could be spelling mistake, etc!
Option left: name:-ksp_monitor (no value)

From dave.mayhem23 at gmail.com  Fri Sep 25 00:34:56 2015
From: dave.mayhem23 at gmail.com (Dave May)
Date: Fri, 25 Sep 2015 07:34:56 +0200
Subject: [petsc-users] Direct inversion methods in parallel
In-Reply-To: <CAGi1ndTrqxGyW-J7c3zxvY=VDDHcvUMWCQwUQxwgq2ZnD9+5Pg@mail.gmail.com>
References: <CAGi1ndTrqxGyW-J7c3zxvY=VDDHcvUMWCQwUQxwgq2ZnD9+5Pg@mail.gmail.com>
Message-ID: <CAJ98EDqH-pmGZO0hUrn8TKn_iCNWQ-zwZ9cnGRP9QiSFL5etDA@mail.gmail.com>

On 25 September 2015 at 07:24, Timoth?e Nicolas <timothee.nicolas at gmail.com>
wrote:

> Hi all, from the manual, I get that the options
>
> -ksp_type preonly -pc_type lu
>
> to solve a problem by direct LU inversion
>
This is doing LU factorization.
The inverse matrix is not assembled explicitly.


> are available only for sequential matrices. Should I conclude that there
> is no method to try a direct inversion of a big problem in parallel ?
>

The packages,
  superlu_dist, mumps and pastix
provided support for parallel LU factorization.
These packages can be installed by petsc'c configure system.


> I plan to use the direct inversion only as a check that my approximation
> to the inverse problem is OK, because so far my algorithm which should work
> is not working at all and I need to debug what is going on. Namely I use an
> approximation to the linear problem using an approximate Schur complement,
> and I want to know if my approximation is false or if from the start my
> matrices are false.
>
> I have tried a direct inversion on one process with the above options for
> a quite small problem (12x12x40 with 8 dof), but it did not work, I suppose
> because of memory limitation (output with log_summary at the end attached
> just in case).
>

>From the output it appears you are running a debug build of petsc.
If you want to see an immediate gain in performance, profile your algorithm
with an optimized build of petsc.
Also, if you want to get better performance from sequential sparse direct
solvers, consider using the packages
  umfpack (or cholmod if the matrix is symmetric)
available from suitesparse.
These libraries are great.
The implementations also leverage multi-threaded BLAS thus they will be
much faster than using petsc default LU.

Cheers
   Dave

Best
>
> Timothee NICOLAS
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150925/042b61ad/attachment.html>

From timothee.nicolas at gmail.com  Fri Sep 25 02:32:09 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Fri, 25 Sep 2015 16:32:09 +0900
Subject: [petsc-users] Direct inversion methods in parallel
In-Reply-To: <CAJ98EDqH-pmGZO0hUrn8TKn_iCNWQ-zwZ9cnGRP9QiSFL5etDA@mail.gmail.com>
References: <CAGi1ndTrqxGyW-J7c3zxvY=VDDHcvUMWCQwUQxwgq2ZnD9+5Pg@mail.gmail.com>
	<CAJ98EDqH-pmGZO0hUrn8TKn_iCNWQ-zwZ9cnGRP9QiSFL5etDA@mail.gmail.com>
Message-ID: <CAGi1ndRFFzgh_OygWgD=Z0d5_M2RKcBONAMi+JWTvZJW9oUDMQ@mail.gmail.com>

Thanks, I will have a look. Regarding the performance, I am just using my
desktop computer here, on the supercomputer, I don't have the issue that it
is compiled with the debugging options. In any case, I am not yet at the
point of optimizing performance

Cheers

Timothee

2015-09-25 14:34 GMT+09:00 Dave May <dave.mayhem23 at gmail.com>:

>
>
> On 25 September 2015 at 07:24, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
>
>> Hi all, from the manual, I get that the options
>>
>> -ksp_type preonly -pc_type lu
>>
>> to solve a problem by direct LU inversion
>>
> This is doing LU factorization.
> The inverse matrix is not assembled explicitly.
>
>
>> are available only for sequential matrices. Should I conclude that there
>> is no method to try a direct inversion of a big problem in parallel ?
>>
>
> The packages,
>   superlu_dist, mumps and pastix
> provided support for parallel LU factorization.
> These packages can be installed by petsc'c configure system.
>
>
>> I plan to use the direct inversion only as a check that my approximation
>> to the inverse problem is OK, because so far my algorithm which should work
>> is not working at all and I need to debug what is going on. Namely I use an
>> approximation to the linear problem using an approximate Schur complement,
>> and I want to know if my approximation is false or if from the start my
>> matrices are false.
>>
>> I have tried a direct inversion on one process with the above options for
>> a quite small problem (12x12x40 with 8 dof), but it did not work, I suppose
>> because of memory limitation (output with log_summary at the end attached
>> just in case).
>>
>
> From the output it appears you are running a debug build of petsc.
> If you want to see an immediate gain in performance, profile your
> algorithm with an optimized build of petsc.
> Also, if you want to get better performance from sequential sparse direct
> solvers, consider using the packages
>   umfpack (or cholmod if the matrix is symmetric)
> available from suitesparse.
> These libraries are great.
> The implementations also leverage multi-threaded BLAS thus they will be
> much faster than using petsc default LU.
>
> Cheers
>    Dave
>
> Best
>>
>> Timothee NICOLAS
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150925/052c0c59/attachment.html>

From zonexo at gmail.com  Fri Sep 25 03:20:42 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 25 Sep 2015 16:20:42 +0800
Subject: [petsc-users] Insert values into matrix using
	MatSetValuesStencil
In-Reply-To: <CAGi1ndRxBT77xKRfPC58ewCz21_N2H16Vh2ZVfcGeq08e+ABQA@mail.gmail.com>
References: <CADSzT4JLACgt_UZvAG+uyDkNaSOzs1t4QAr0T4DozWN--8Fe+w@mail.gmail.com>
	<CAGi1ndQLKN_+cUBchLD8SAqDn8QO46TQf+L69rjqqCLkXaX1UA@mail.gmail.com>
	<55DD3CC1.5070801@gmail.com>
	<CAGi1ndR78hB8M9_ybho-favYfdJnsissE_A45V1hFYZTQ0AC0w@mail.gmail.com>
	<55DD4869.2000006@gmail.com>
	<CAGi1ndQmCWRbx5ijVc-vc96RaOGuSXwoXz43FRpjyfBs8e1O7Q@mail.gmail.com>
	<56025F54.3040604@gmail.com>
	<CAGi1ndR_fFN=qyycB20nML-j1n=sQbMZKWqFfnSmqSS7P84=8Q@mail.gmail.com>
	<CAGi1ndRp69mEq8VnK86Uc8HYvZ6b5zXjYdKvtqkw3niC2ew9-Q@mail.gmail.com>
	<560264C4.70205@gmail.com>
	<CAGi1ndQ9gCf-_u9H0r=o+S8z0P4pWU2LeQLGffb_wOSt-dx8EQ@mail.gmail.com>
	<56026D25.2010105@gmail.com>
	<CAGi1ndRxBT77xKRfPC58ewCz21_N2H16Vh2ZVfcGeq08e+ABQA@mail.gmail.com>
Message-ID: <560503DA.3090605@gmail.com>

Hi Timoth?e,

I got the error sorted out. I was inserting into the wrong matrix. But 
the key error was the defn of row and col.

Thank you for pointing it out.

Yours sincerely,

TAY wee-beng

On 23/9/2015 5:25 PM, Timoth?e Nicolas wrote:
> Hum, sorry, I don't know. I asked you to provide the definitions of 
> start and end values of i,j,k, because I was concerned whether you may 
> mess up the boundaries. Especially because you seem to treat x 
> differently from y and z. You have the problem also on only 1 process ?
>
> 2015-09-23 18:13 GMT+09:00 TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>>:
>
>     Hi Timoth?e,
>
>     Maybe I can send you part of it 1st. I'm trying to pinpoint why my
>     matrix using MatView shows zero for a lot of the values
>
>     For i=1,j=1,k=1,
>
>     It should be :
>
>     Mat Object: 1 MPI processes
>       type: seqaij
>     row 0: (0, 2)  (12, -2)
>
>     but now it's:
>
>     row 0: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7,
>     0)  (8, 0)  (12, 0)  (13, 0)  (14, 0)  (24, 0) (25, 0)  (26, 0) 
>     (96, 0)  (97, 0)  (98, 0)  (192, 0) (193, 0)  (194, 0)
>
>     I used:
>
>     /call
>     DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,&//
>     //
>     //size_z,1,PETSC_DECIDE,PETSC_DECIDE,3,stencil_width,lx,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da_uvw_ast,ierr)//
>     //
>     //call DMSetMatType(da_uvw_ast,MATAIJ,ierr)//
>     ////
>     //    call DMCreateMatrix(da_uvw_ast,A_semi_xyz,ierr)//
>     //
>     //    call MatSetFromOptions(A_semi_xyz,ierr)//
>     //
>     //i = 1, j = 1, k = 1//
>     //
>     //row(MatStencil_i,1) = i - 1//
>     ////
>     //                    row(MatStencil_j,1) = j - 1//
>     ////
>     //                    row(MatStencil_k,1) = k - 1//
>     ////
>     //                    row(MatStencil_c,1) = 0//
>     ////
>     //                    col(MatStencil_i,1) = i - 1//
>     ////
>     //                    col(MatStencil_j,1) = j - 1//
>     ////
>     //                    col(MatStencil_k,1) = k - 1//
>     ////
>     //                    col(MatStencil_c,1) = 0//
>     ////
>     //                    value_insert(1) = 2.//
>     //
>     //                    call
>     MatSetValuesStencil(A_mat,ione,row,ione,col(:,1),value_insert(1),INSERT_VALUES,ierr)//
>     //
>     //                    col(MatStencil_i,2) = i - 1 !north//
>     ////
>     //                    col(MatStencil_j,2) = j + 1 - 1//
>     ////
>     //                    col(MatStencil_k,2) = k - 1//
>     ////
>     //                    col(MatStencil_c,2) = 0//
>     ////
>     //                    value_insert(2) = -2.//
>     //
>     //                    call
>     MatSetValuesStencil(A_mat,ione,row,ione,col(:,2),value_insert(2),INSERT_VALUES,ierr)/
>
>     Thank you
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>     On 23/9/2015 4:45 PM, Timoth?e Nicolas wrote:
>>     Yes, I had understood that, I am doing the same, but with 8 dof.
>>     This does not change the declaration for row and col.
>>
>>     Can you send (i) the commands you use to create the DMDA, (ii)
>>     the commands to create the matrix and (iii) those for the
>>     definitions of ksta2,kend2,jsta2,jend2,size_x ?
>>
>>     Timoth?e
>>
>>     2015-09-23 17:37 GMT+09:00 TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>>:
>>
>>         Hi Timoth?e,
>>
>>         The matrix is created with 3 dof - u,v,w. So for each i,j,k,
>>         there are 3 values. Actually I got 3 eqns from the u,v,w
>>         momentum eqns. They are not coupled together so I can also
>>         solve them individually. But I was told it's faster to group
>>         them together. The global indices is something like this:
>>
>>         i, j, k, ijk global indices, dof
>>         1 1 1 1 u
>>         1 1 1 2 v
>>         1 1 1 3 w
>>         2 1 1 4 u
>>
>>         ...
>>
>>         Hope it's clearer now.
>>
>>         Ok, I changed the stencil, it's now working.
>>
>>         Thank you
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>         On 23/9/2015 4:22 PM, Timoth?e Nicolas wrote:
>>>         Can you also tell how you created the matrix ? Just in case
>>>         you created it with the 1 dof DMDA, it would not work if you
>>>         try to input values at places where it is not allocated
>>>         (which could explain the error message)
>>>
>>>         2015-09-23 17:18 GMT+09:00 Timoth?e Nicolas
>>>         <timothee.nicolas at gmail.com
>>>         <mailto:timothee.nicolas at gmail.com>>:
>>>
>>>             The first thing that strikes me is your definition of
>>>             the stencils
>>>
>>>             */MatStencil  :: row(6,1),col(6,7)/**/
>>>
>>>             /*
>>>             Why is it not defined with
>>>
>>>             */MatStencil  :: row(4,1),col(4,7)/**/
>>>             /*
>>>             instead ?
>>>
>>>             Where does the 6 come from ?
>>>
>>>             Timoth?e
>>>
>>>
>>>             2015-09-23 17:14 GMT+09:00 TAY wee-beng
>>>             <zonexo at gmail.com <mailto:zonexo at gmail.com>>:
>>>
>>>                 Hi,
>>>
>>>                 I have successfully used MatSetValuesStencil to
>>>                 insert values into a Poisson eqn matrix which has 1
>>>                 DOF (pressure). Now I'm trying to insert values in a
>>>                 momentum eqn matrix which has 3 DOF (u,v,w)
>>>
>>>                 However, I got the error:
>>>
>>>                 /*[0]PETSC ERROR: --------------------- Error
>>>                 Message ----------------------------*//*
>>>                 *//*----------------------------------*//*
>>>                 *//*[0]PETSC ERROR: Argument out of range*//*
>>>                 *//*[0]PETSC ERROR: Inserting a new nonzero at
>>>                 (111,5) in the matrix*//*
>>>                 *//*[0]PETSC ERROR: See
>>>                 http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>                 for trou*//*
>>>                 *//*ble shooting.*/
>>>
>>>                 I wonder what's wrong.  For the momentum eqn, for
>>>                 each DOF, at at node (dof,i,j,k), I have coupling i
>>>                 +/- 1, j +/- 1 and k +/- 1.
>>>
>>>                 The error happens at 111,5, which corresponds to i =
>>>                 2, j = 2, k = 2, which is an internal node.
>>>
>>>                 Here's part of my code below. Hope someone can help.
>>>                 Thanks!
>>>                 */
>>>                 /**/PetscScalar :: value_insert(7)/**/
>>>                 /**/
>>>                 /**/MatStencil :: row(6,1),col(6,7)/**/
>>>                 /**/
>>>                 /**/ione = 1;   iseven = 7/**/
>>>                 /**/
>>>                 /**/if (cell_type == 'u') then/**/
>>>                 /**/
>>>                 /**/offset = 1/**/
>>>                 /**//**/
>>>                 /**/else if (cell_type == 'v') then/**/
>>>                 /**/
>>>                 /**/offset = 2/**/
>>>                 /**//**/
>>>                 /**/else if (cell_type == 'w') then/**/
>>>                 /**/
>>>                 /**/offset = 3/**/
>>>                 /**//**/
>>>                 /**/end if/**/
>>>                 /**/
>>>                 /**/do k=ksta2,kend2/**/
>>>                 /**/
>>>                 /**/do j = jsta2,jend2/**/
>>>                 /**/
>>>                 /**/    do i=2,size_x-1/**/
>>>                 /**//**/
>>>                 /**/row(MatStencil_i,1) = i - 1/**/
>>>                 /**//**/
>>>                 /**/row(MatStencil_j,1) = j - 1/**/
>>>                 /**//**/
>>>                 /**/row(MatStencil_k,1) = k - 1/**/
>>>                 /**//**/
>>>                 /**/row(MatStencil_c,1) = offset - 1/**/
>>>                 /**//**/
>>>                 /**/value_insert = 0.d0/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_i,3) = i + 1 - 1 !east/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_j,3) = j - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_k,3) = k - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_c,3) = offset - 1/**/
>>>                 /**//**/
>>>                 /**/value_insert(3) = -(
>>>                 1./(cell_x(i)%pd_E+cell_x(i+1)%pd_W))*(c_yz(j,k)%fc_E)*inv_Re/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_i,5) = i - 1 - 1 !west/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_j,5) = j - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_k,5) = k - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_c,5) = offset - 1/**/
>>>                 /**//**/
>>>                 /**/value_insert(5) = -(
>>>                 1./(cell_x(i)%pd_W+cell_x(i-1)%pd_E))*(c_yz(j,k)%fc_E)*inv_Re/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_i,2) = i - 1 !north/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_j,2) = j + 1 - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_k,2) = k - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_c,2) = offset - 1/**/
>>>                 /**//**/
>>>                 /**/value_insert(2) = -(
>>>                 1./(cell_y(j)%pd_N+cell_y(j+1)%pd_S))*(c_zx(i,k)%fc_N)*inv_Re/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_i,4) = i - 1 !south/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_j,4) = j - 1 - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_k,4) = k - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_c,4) = offset - 1/**/
>>>                 /**//**/
>>>                 /**/value_insert(4) = -(
>>>                 1./(cell_y(j)%pd_S+cell_y(j-1)%pd_N))*(c_zx(i,k)%fc_N)*inv_Re/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_i,6) = i - 1 !front/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_j,6) = j - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_k,6) = k + 1 - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_c,6) = offset - 1/**/
>>>                 /**//**/
>>>                 /**/value_insert(6) = -(
>>>                 1./(cell_z(k)%pd_F+cell_z(k+1)%pd_B))*(c_xy(i,j)%fc_F)*inv_Re/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_i,7) = i - 1 !back/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_j,7) = j - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_k,7) = k - 1 - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_c,7) = offset - 1/**/
>>>                 /**//**/
>>>                 /**/value_insert(7) = -(
>>>                 1./(cell_z(k)%pd_B+cell_z(k-1)%pd_F))*(c_xy(i,j)%fc_F)*inv_Re/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_i,1) = i - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_j,1) = j - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_k,1) = k - 1/**/
>>>                 /**//**/
>>>                 /**/col(MatStencil_c,1) = offset - 1/**/
>>>                 /**//**/
>>>                 /**/value_insert(1) = 2.*c(i,j,k)%vol/del_t -
>>>                 (value_insert(2)+value_insert(3)+value_insert(4)+value_insert(5)+value_insert(6)+value_insert(7))/**/
>>>                 /**//**/
>>>                 /**/call
>>>                 MatSetValuesStencil(A_semi_xyz,ione,row,iseven,col,value_insert,INSERT_VALUES,ierr)/**/
>>>                 /**//**/
>>>                 /**/end do/**/
>>>                 /**//**/
>>>                 /**/end do/**/
>>>                 /**/
>>>                 /**/end do /*
>>>
>>>                 Thank you
>>>
>>>                 Yours sincerely,
>>>
>>>                 TAY wee-beng
>>>
>>>
>>>
>>>
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150925/277267ca/attachment-0001.html>

From snakexf at gmail.com  Fri Sep 25 04:16:00 2015
From: snakexf at gmail.com (Feng Xing)
Date: Fri, 25 Sep 2015 11:16:00 +0200
Subject: [petsc-users] reuse matrix aij structure
Message-ID: <364393C2-F96C-4124-9A55-E9B1F3DAA872@gmail.com>

Hello everyone,

I want to solve many times A*x=b, where the non-zero structure of A (aij) is always same, but its values change. I would like to ask if it is possible to call assembly_begin/end only one time, then change the values of A each time. 

I find it is possible for a vector(Vec) through VecGetArray. It returns a pointer. Is there similar way for matrix? Thank you very much.

Regards,

Feng Xing
Postdoc at Inria



From knepley at gmail.com  Fri Sep 25 07:28:45 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 25 Sep 2015 07:28:45 -0500
Subject: [petsc-users] reuse matrix aij structure
In-Reply-To: <364393C2-F96C-4124-9A55-E9B1F3DAA872@gmail.com>
References: <364393C2-F96C-4124-9A55-E9B1F3DAA872@gmail.com>
Message-ID: <CAMYG4GnAc4wLchSSpvoE1TH3C7rG1++jowUfA2Ntr2vVWA2nCQ@mail.gmail.com>

On Fri, Sep 25, 2015 at 4:16 AM, Feng Xing <snakexf at gmail.com> wrote:

> Hello everyone,
>
> I want to solve many times A*x=b, where the non-zero structure of A (aij)
> is always same, but its values change. I would like to ask if it is
> possible to call assembly_begin/end only one time, then change the values
> of A each time.
>

You need to call assembly when changing values, but it should be fast.

   Matt


> I find it is possible for a vector(Vec) through VecGetArray. It returns a
> pointer. Is there similar way for matrix? Thank you very much.
>
> Regards,
>
> Feng Xing
> Postdoc at Inria
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150925/20f6b612/attachment.html>

From bsmith at mcs.anl.gov  Fri Sep 25 11:12:17 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 25 Sep 2015 11:12:17 -0500
Subject: [petsc-users] reuse matrix aij structure
In-Reply-To: <CAMYG4GnAc4wLchSSpvoE1TH3C7rG1++jowUfA2Ntr2vVWA2nCQ@mail.gmail.com>
References: <364393C2-F96C-4124-9A55-E9B1F3DAA872@gmail.com>
	<CAMYG4GnAc4wLchSSpvoE1TH3C7rG1++jowUfA2Ntr2vVWA2nCQ@mail.gmail.com>
Message-ID: <76347D85-BE02-4132-B9CB-5A880BDB9B63@mcs.anl.gov>


> On Sep 25, 2015, at 7:28 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Sep 25, 2015 at 4:16 AM, Feng Xing <snakexf at gmail.com> wrote:
> Hello everyone,
> 
> I want to solve many times A*x=b, where the non-zero structure of A (aij) is always same, but its values change. I would like to ask if it is possible to call assembly_begin/end only one time, then change the values of A each time.
> 
> You need to call assembly when changing values, but it

   will 

> should be fast.
> 
>    Matt
>  
> I find it is possible for a vector(Vec) through VecGetArray. It returns a pointer. Is there similar way for matrix? Thank you very much.
> 
> Regards,
> 
> Feng Xing
> Postdoc at Inria
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From timothee.nicolas at gmail.com  Mon Sep 28 02:53:48 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Mon, 28 Sep 2015 16:53:48 +0900
Subject: [petsc-users] KSP step not accelerating despite good
	preconditioning in SNES
Message-ID: <CAGi1ndQHEUKmhiXLkJG3D9HAbCbqxisSLHWbgKwDJ5kfwd5TfQ@mail.gmail.com>

Hi all,

I have something strange in my application and I don't know what could
cause this. I am trying to do an implicit MHD problem and I thought I
finally figured out the preconditioner step, but somehow I don't get the
expected result, not even close.

For the preconditioning, I am using an approximate Schur complement, which
requires two relatively easy KSP inversions at each preconditioner
application. I apply this algorithm directly to the result function at the
end of the routine FormFunction. I have checked that the approximation to
the inversion of the Jacobian is good, in the sense that when I multiply
the preconditioned vector by the *total* Jacobian matrix, I indeed recover
almost the initial unpreconditioned vector. Also, I know that my Jacobian
matrix is correct, because (i) I have checked manually that F(X + dX) ~
F(X) + J * dX and (ii) when I don't use -snes_mf and use the provided
Jacobian matrix the result is pretty much equivalent to using -snes_mf.

In my understanding, this means that what I effectively feed to SNES at the
end of my FormFunction routine is a good approximation to J^(-1) F. As a
result, I naturally expect that the number of KSP iterations necessary to
achieve one SNES iteration be drastically reduced. However, I observe
virtually no change whatsoever in the number of iterations.

Any thoughts about what I could be missing ? Maybe I forgot to set a SNES
or KSP option somewhere ? I can send pieces of code if needs be.

Best

Timothee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150928/b04c978f/attachment.html>

From knepley at gmail.com  Mon Sep 28 07:17:35 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 28 Sep 2015 07:17:35 -0500
Subject: [petsc-users] KSP step not accelerating despite good
 preconditioning in SNES
In-Reply-To: <CAGi1ndQHEUKmhiXLkJG3D9HAbCbqxisSLHWbgKwDJ5kfwd5TfQ@mail.gmail.com>
References: <CAGi1ndQHEUKmhiXLkJG3D9HAbCbqxisSLHWbgKwDJ5kfwd5TfQ@mail.gmail.com>
Message-ID: <CAMYG4Gkw--6We9=s8_fxLHoy1AuHudyyENu6xbg=18Gbsgmy5Q@mail.gmail.com>

On Mon, Sep 28, 2015 at 2:53 AM, Timoth?e Nicolas <
timothee.nicolas at gmail.com> wrote:

> Hi all,
>
> I have something strange in my application and I don't know what could
> cause this. I am trying to do an implicit MHD problem and I thought I
> finally figured out the preconditioner step, but somehow I don't get the
> expected result, not even close.
>
> For the preconditioning, I am using an approximate Schur complement, which
> requires two relatively easy KSP inversions at each preconditioner
> application. I apply this algorithm directly to the result function at the
> end of the routine FormFunction. I have checked that the approximation to
> the inversion of the Jacobian is good, in the sense that when I multiply
> the preconditioned vector by the *total* Jacobian matrix, I indeed
> recover almost the initial unpreconditioned vector. Also, I know that my
> Jacobian matrix is correct, because (i) I have checked manually that F(X +
> dX) ~ F(X) + J * dX and (ii) when I don't use -snes_mf and use the provided
> Jacobian matrix the result is pretty much equivalent to using -snes_mf.
>
> In my understanding, this means that what I effectively feed to SNES at
> the end of my FormFunction routine is a good approximation to J^(-1) F. As
> a result, I naturally expect that the number of KSP iterations necessary to
> achieve one SNES iteration be drastically reduced. However, I observe
> virtually no change whatsoever in the number of iterations.
>
> Any thoughts about what I could be missing ? Maybe I forgot to set a SNES
> or KSP option somewhere ? I can send pieces of code if needs be.
>

It sounds like you are putting this in the wrong place. If you have the
action of a good preconditioner for the Jacobian, then you
should use a PCSHELL and pass it to the KSP. It does not belong in the
FormFunction.

  Thanks,

     Matt


> Best
>
> Timothee
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150928/6dcb8ab9/attachment.html>

From timothee.nicolas at gmail.com  Mon Sep 28 07:21:00 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Mon, 28 Sep 2015 21:21:00 +0900
Subject: [petsc-users] KSP step not accelerating despite good
 preconditioning in SNES
In-Reply-To: <CAMYG4Gkw--6We9=s8_fxLHoy1AuHudyyENu6xbg=18Gbsgmy5Q@mail.gmail.com>
References: <CAGi1ndQHEUKmhiXLkJG3D9HAbCbqxisSLHWbgKwDJ5kfwd5TfQ@mail.gmail.com>
	<CAMYG4Gkw--6We9=s8_fxLHoy1AuHudyyENu6xbg=18Gbsgmy5Q@mail.gmail.com>
Message-ID: <CAGi1ndRNEdFHeRtrLL5i6eAxQoQJ4ygqhsg9ORKnXjVXKGVj+A@mail.gmail.com>

Thanks a lot, I expected something like that. I will look in this direction.

Best

Timoth?e

2015-09-28 21:17 GMT+09:00 Matthew Knepley <knepley at gmail.com>:

> On Mon, Sep 28, 2015 at 2:53 AM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
>
>> Hi all,
>>
>> I have something strange in my application and I don't know what could
>> cause this. I am trying to do an implicit MHD problem and I thought I
>> finally figured out the preconditioner step, but somehow I don't get the
>> expected result, not even close.
>>
>> For the preconditioning, I am using an approximate Schur complement,
>> which requires two relatively easy KSP inversions at each preconditioner
>> application. I apply this algorithm directly to the result function at the
>> end of the routine FormFunction. I have checked that the approximation to
>> the inversion of the Jacobian is good, in the sense that when I multiply
>> the preconditioned vector by the *total* Jacobian matrix, I indeed
>> recover almost the initial unpreconditioned vector. Also, I know that my
>> Jacobian matrix is correct, because (i) I have checked manually that F(X +
>> dX) ~ F(X) + J * dX and (ii) when I don't use -snes_mf and use the provided
>> Jacobian matrix the result is pretty much equivalent to using -snes_mf.
>>
>> In my understanding, this means that what I effectively feed to SNES at
>> the end of my FormFunction routine is a good approximation to J^(-1) F. As
>> a result, I naturally expect that the number of KSP iterations necessary to
>> achieve one SNES iteration be drastically reduced. However, I observe
>> virtually no change whatsoever in the number of iterations.
>>
>> Any thoughts about what I could be missing ? Maybe I forgot to set a SNES
>> or KSP option somewhere ? I can send pieces of code if needs be.
>>
>
> It sounds like you are putting this in the wrong place. If you have the
> action of a good preconditioner for the Jacobian, then you
> should use a PCSHELL and pass it to the KSP. It does not belong in the
> FormFunction.
>
>   Thanks,
>
>      Matt
>
>
>> Best
>>
>> Timothee
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150928/f0f857de/attachment.html>

From brbfernandes at utexas.edu  Tue Sep 29 00:54:02 2015
From: brbfernandes at utexas.edu (Bruno Ramon Batista Fernande)
Date: Tue, 29 Sep 2015 00:54:02 -0500
Subject: [petsc-users] Error when compiling petsc on windows
Message-ID: <CAEpUo3gfkkRLcNya8pxck4gNyVWHLtFgnDSGrFKbmtcixbE9pQ@mail.gmail.com>

Dear petsc users,

I have been trying to use use PETSC on my project on windows. I have
followed all steps and compiled petsc on windows using cygwin and cmake.
Then I included it on my visual studio project with all other dependencies.
After fixing several errors I got stuck with this one:

Error 1 error LNK2019: unresolved external symbol __imp_GetUserNameA
referenced in function PetscGetUserName libpetsc.lib(fuser.o)

Anyone has any Idea of which library I should include to solve such
problem?

I am using windows 8 machine with visual studio 2013 and Intel fortran
2013, MS MPI, metis, parmetis, and mkl. All libraries are compiled to x64
and the platform on the VS project was set to x64. I have also activated
the use of common windows libraries on the VS project to solve other
problems. This is the configure I used:

./configure --with-cc='win32fe cl' --with-fc='win32fe ifort'
--prefix=/cygdrive/c/Installed/petsc-opt/ --PETSC_ARCH=windows-intel-opt
--with-parmetis=1
--with-parmetis-include=/cygdrive/c/Installed/Parmetis/include_all/
--with-parmetis-lib=\[/cygdrive/c/Installed/Parmetis/lib/parmetis.lib\]
--with-metis=1
--with-metis-include=/cygdrive/c/Installed/Parmetis/include_all/
--with-metis-lib=\[/cygdrive/c/Installed/Parmetis/lib/metis.lib\]
--with-mpi-include=[/cygdrive/c/Installed/MPI/Include/,/cygdrive/c/Installed/MPI/Include/x64/]
--with-mpi-lib=[/cygdrive/c/Installed/MPI/Lib/x64/msmpi.lib,/cygdrive/c/Installed/MPI/Lib/x64/msmpifec.lib]
--with-blas-lapack-lib=[/cygdrive/c/Installed/mkl/lib/intel64/mkl_intel_lp64.lib,/cygdrive/c/Installed/mkl/lib/intel64/mkl_core.lib,/cygdrive/c/Installed/mkl/lib/intel64/mkl_sequential.lib]
--with-scalapack-include=/cygdrive/c/Installed/mkl/include
--with-scalapack-lib=[/cygdrive/c/Installed/mkl/lib/intel64/mkl_scalapack_lp64.lib,/cygdrive/c/Installed/mkl/lib/intel64/mkl_blacs_msmpi_lp64.lib]
--with-fortran-interfaces=1 --with-debugging=0

regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/d4dfeb76/attachment.html>

From zonexo at gmail.com  Tue Sep 29 01:27:47 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Tue, 29 Sep 2015 14:27:47 +0800
Subject: [petsc-users] Numbering in MatView
Message-ID: <560A2F63.9010409@gmail.com>

Hi,

When I use MatView to output to an ASCII file, it gives:

Mat Object: 2 MPI processes
   type: mpiaij
row 0: (0, 1)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)  
(8, 0)  (12, -1)  (13, 0)  (14, 0)  (24, 0)  (25, 0)  (26, 0) (96, 0)  
(97, 0)  (98, 0)  (192, 0)  (193, 0)  (194, 0)
row 1: (0, 0)  (1, 13.4413)  (2, 0)  (3, 0)  (4, -0.00187601)  (5, 0)  
(6, 0)  (7, 0)  (8, 0)  (12, 0)  (13, -0.000154087)  (14, 0) (24, 0)  
(25, 0)  (26, 0)  (96, 0)  (97, -1.81928e-05)  (98, 0) (192, 0)  (193, 
0)  (194, 0)
row 2: (0, 0)  (1, 0)  (2, 1)  (3, 0)  (4, 0...

So how is the row numbered?

Does it uses the natural ordering or the PETSc ordering?

It seems to be the natural ordering since it doesn't change when I use 
1,2 or 4 cpus, is that so?

-- 
Thank you

Yours sincerely,

TAY wee-beng


From jychang48 at gmail.com  Tue Sep 29 03:02:10 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 29 Sep 2015 02:02:10 -0600
Subject: [petsc-users] multi-grid for anisotropy
Message-ID: <DE567C75-C756-4476-B96B-D5CD195370B5@gmail.com>

Hi all,

I think I have asked a previous question related to this, and have seen other threads about related issues, but say I have this anisotropic dispersion tensor:

D = (D_m + alpha_L*norm(v))*Identity + (alpha_L - alpha_T)*outer(v,v)/norm(v)

The anisotropy of this tensor comes from the ratio of alpha_L/alpha_T and velocity v, and is mesh independent. That said, can tuning the multi-grid preconditioners parameters of GAMG/HYPRE/etc help prevent things like negative concentrations and violations of the discrete maximum principles?

Thanks,

From bsmith at mcs.anl.gov  Tue Sep 29 08:20:37 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 29 Sep 2015 08:20:37 -0500
Subject: [petsc-users] Numbering in MatView
In-Reply-To: <560A2F63.9010409@gmail.com>
References: <560A2F63.9010409@gmail.com>
Message-ID: <8CDF056F-F4F7-4F8F-ACA6-37B933F04F89@mcs.anl.gov>


  If the matrix came from a DMDA then it is displayed in the "natural" ordering. Otherwise it is displayed in whatever parallel ordering it is using.

   Barry

> On Sep 29, 2015, at 1:27 AM, TAY wee-beng <zonexo at gmail.com> wrote:
> 
> Hi,
> 
> When I use MatView to output to an ASCII file, it gives:
> 
> Mat Object: 2 MPI processes
>  type: mpiaij
> row 0: (0, 1)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)  (8, 0)  (12, -1)  (13, 0)  (14, 0)  (24, 0)  (25, 0)  (26, 0) (96, 0)  (97, 0)  (98, 0)  (192, 0)  (193, 0)  (194, 0)
> row 1: (0, 0)  (1, 13.4413)  (2, 0)  (3, 0)  (4, -0.00187601)  (5, 0)  (6, 0)  (7, 0)  (8, 0)  (12, 0)  (13, -0.000154087)  (14, 0) (24, 0)  (25, 0)  (26, 0)  (96, 0)  (97, -1.81928e-05)  (98, 0) (192, 0)  (193, 0)  (194, 0)
> row 2: (0, 0)  (1, 0)  (2, 1)  (3, 0)  (4, 0...
> 
> So how is the row numbered?
> 
> Does it uses the natural ordering or the PETSc ordering?
> 
> It seems to be the natural ordering since it doesn't change when I use 1,2 or 4 cpus, is that so?
> 
> -- 
> Thank you
> 
> Yours sincerely,
> 
> TAY wee-beng
> 


From knepley at gmail.com  Tue Sep 29 09:13:12 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 29 Sep 2015 09:13:12 -0500
Subject: [petsc-users] multi-grid for anisotropy
In-Reply-To: <DE567C75-C756-4476-B96B-D5CD195370B5@gmail.com>
References: <DE567C75-C756-4476-B96B-D5CD195370B5@gmail.com>
Message-ID: <CAMYG4G=nWVAi+p8q=6mkuYfni0+qXmH-ZghzEoJY0Zz7do1O6g@mail.gmail.com>

On Tue, Sep 29, 2015 at 3:02 AM, Justin Chang <jychang48 at gmail.com> wrote:

> Hi all,
>
> I think I have asked a previous question related to this, and have seen
> other threads about related issues, but say I have this anisotropic
> dispersion tensor:
>
> D = (D_m + alpha_L*norm(v))*Identity + (alpha_L -
> alpha_T)*outer(v,v)/norm(v)
>
> The anisotropy of this tensor comes from the ratio of alpha_L/alpha_T and
> velocity v, and is mesh independent. That said, can tuning the multi-grid
> preconditioners parameters of GAMG/HYPRE/etc help prevent things like
> negative concentrations and violations of the discrete maximum principles?
>

In my view (Barry and Jed can tell me I am wrong), this is the job of the
interpolator. You already guarantee the bound
in your smoothers, so this is where is gets messed up. You can certainly
use your own interpolators in GMG. GAMG
is harder since it makes the aggregates, and I do not understand Hypre.

The way I imagine doing this is to setup the interpolation as a saddle
point problem, where the thing you want to enforce
is a constraint, so the interpolation is a solve instead of jsut a matrix.
This is exactly what I did for my Phd when we transferred
fluid-structure solutions from one mesh to another.

  Thanks,

     Matt


> Thanks,

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/f26129c4/attachment.html>

From jed at jedbrown.org  Tue Sep 29 09:17:36 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 29 Sep 2015 08:17:36 -0600
Subject: [petsc-users] multi-grid for anisotropy
In-Reply-To: <DE567C75-C756-4476-B96B-D5CD195370B5@gmail.com>
References: <DE567C75-C756-4476-B96B-D5CD195370B5@gmail.com>
Message-ID: <87oagl5myn.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:

> Hi all,
>
> I think I have asked a previous question related to this, and have
> seen other threads about related issues, but say I have this
> anisotropic dispersion 

Diffusion and dispersion are different phenomena.  Sounds like you mean
diffusion.

> tensor:
>
> D = (D_m + alpha_L*norm(v))*Identity + (alpha_L - alpha_T)*outer(v,v)/norm(v)
>
> The anisotropy of this tensor comes from the ratio of alpha_L/alpha_T and velocity v, and is mesh independent. That said, can tuning the multi-grid preconditioners parameters of GAMG/HYPRE/etc help prevent things like negative concentrations and violations of the discrete maximum principles?

Hell no!  Tuning an algebraic solver doesn't fix a non-monotone discretization.

That said, the linear solve could converge "from below", in which case
you might want to perform a cleaning step that fixes values by less than
the algebraic solver tolerance.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/f54e8214/attachment.pgp>

From gforsyth at gwu.edu  Tue Sep 29 10:28:41 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Tue, 29 Sep 2015 11:28:41 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
Message-ID: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>

Hi all,

I've been having some trouble with what should be a relatively simple
update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1

I'm getting indefinite PC errors for a simple lid-driven cavity test
problem, 32x32 at Re 100

Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set the
null space.  This is for a 2D Poisson system with no immersed boundary and
so the null space is the constant vector.

  MatNullSpace nsp;
  ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL, &nsp);
CHKERRQ(ierr);
  ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
  ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);


And then setup the KSP with

  ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
  ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
  ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
  ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
  ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
  ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
  ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);


The matrix QTBNQ does not change, only the rhs of the system is updated.

We run this with `-pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1`
and everything seems to work as expected.

Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to

  ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);

and the same code diverges after 1 timestep and returns a -8
KSP_DIVERGED_INDEFINITE_PC

This is weird, especially because if we change nsmooths to 2, it runs for
264 timesteps and the returns the same error.  But we have explicitly set
KSPSetReusePreconditioner so it should be using the same PC, right?

Change nsmooths to 3 and it again diverges after 1 timestep.

Change nsmooths to 4 and it runs to completion.

It seems like either gamg's behavior has changed, or that KSPSetNullSpace
was doing something implicitly that we now need to do explicitly in
addition to MatSetNullSpace?

Thanks,
Gil Forsyth
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/0756aa3a/attachment-0001.html>

From knepley at gmail.com  Tue Sep 29 10:42:29 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 29 Sep 2015 10:42:29 -0500
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
Message-ID: <CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>

On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:

> Hi all,
>
> I've been having some trouble with what should be a relatively simple
> update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>
> I'm getting indefinite PC errors for a simple lid-driven cavity test
> problem, 32x32 at Re 100
>
> Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set the
> null space.  This is for a 2D Poisson system with no immersed boundary and
> so the null space is the constant vector.
>
>   MatNullSpace nsp;
>   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL, &nsp);
> CHKERRQ(ierr);
>   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>
>
Clearly this has to happen in the reverse order, since ksp2 would not be
created yet.

For questions about solvers, we HAVE to see the complete output of
-ksp_view so we
know what we are dealing with. Its also nice to have
-ksp_monitor_true_residual -ksp_converged_reason

  Matt


> And then setup the KSP with
>
>   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>
>
> The matrix QTBNQ does not change, only the rhs of the system is updated.
>
> We run this with `-pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1`
> and everything seems to work as expected.
>
> Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>
>   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>
> and the same code diverges after 1 timestep and returns a -8
> KSP_DIVERGED_INDEFINITE_PC
>
> This is weird, especially because if we change nsmooths to 2, it runs for
> 264 timesteps and the returns the same error.  But we have explicitly set
> KSPSetReusePreconditioner so it should be using the same PC, right?
>
> Change nsmooths to 3 and it again diverges after 1 timestep.
>
> Change nsmooths to 4 and it runs to completion.
>
> It seems like either gamg's behavior has changed, or that KSPSetNullSpace
> was doing something implicitly that we now need to do explicitly in
> addition to MatSetNullSpace?
>
> Thanks,
> Gil Forsyth
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/4f3f2871/attachment.html>

From gforsyth at gwu.edu  Tue Sep 29 10:53:54 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Tue, 29 Sep 2015 11:53:54 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
Message-ID: <CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>

On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>
>> Hi all,
>>
>> I've been having some trouble with what should be a relatively simple
>> update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>
>> I'm getting indefinite PC errors for a simple lid-driven cavity test
>> problem, 32x32 at Re 100
>>
>> Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set the
>> null space.  This is for a 2D Poisson system with no immersed boundary and
>> so the null space is the constant vector.
>>
>>   MatNullSpace nsp;
>>   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
>> &nsp); CHKERRQ(ierr);
>>   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>
>>
> Clearly this has to happen in the reverse order, since ksp2 would not be
> created yet.
>
> For questions about solvers, we HAVE to see the complete output of
> -ksp_view so we
> know what we are dealing with. Its also nice to have
> -ksp_monitor_true_residual -ksp_converged_reason
>
>   Matt
>

Yes -- sorry, those are both in inline files and are called in the reverse
order that I wrote them out.

I've attached the output of

$PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
-poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1 -poisson_ksp_view
-poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
kspview.log



>
>
>> And then setup the KSP with
>>
>>   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>>   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>>   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>
>>
>> The matrix QTBNQ does not change, only the rhs of the system is updated.
>>
>> We run this with `-pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths
>> 1` and everything seems to work as expected.
>>
>> Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>
>>   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>
>> and the same code diverges after 1 timestep and returns a -8
>> KSP_DIVERGED_INDEFINITE_PC
>>
>> This is weird, especially because if we change nsmooths to 2, it runs for
>> 264 timesteps and the returns the same error.  But we have explicitly set
>> KSPSetReusePreconditioner so it should be using the same PC, right?
>>
>> Change nsmooths to 3 and it again diverges after 1 timestep.
>>
>> Change nsmooths to 4 and it runs to completion.
>>
>> It seems like either gamg's behavior has changed, or that KSPSetNullSpace
>> was doing something implicitly that we now need to do explicitly in
>> addition to MatSetNullSpace?
>>
>> Thanks,
>> Gil Forsyth
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/bd4dc688/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kspview.log
Type: text/x-log
Size: 17525 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/bd4dc688/attachment-0001.bin>

From bsmith at mcs.anl.gov  Tue Sep 29 11:37:42 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 29 Sep 2015 11:37:42 -0500
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
Message-ID: <BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>


   This can't work. You can't use a GMRES inside a CG.   Try changing to -poisson_mg_coarse_ksp_type preonly

KSP Object:(poisson_) 1 MPI processes
  type: cg

KSP Object:    (poisson_mg_coarse_)     1 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=1, initial guess is zero


> On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> 
> 
> On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> Hi all, 
> 
> I've been having some trouble with what should be a relatively simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
> 
> I'm getting indefinite PC errors for a simple lid-driven cavity test problem, 32x32 at Re 100
> 
> Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set the null space.  This is for a 2D Poisson system with no immersed boundary and so the null space is the constant vector.
> 
>   MatNullSpace nsp;
>   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL, &nsp); CHKERRQ(ierr);
>   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
> 
> Clearly this has to happen in the reverse order, since ksp2 would not be created yet.
> 
> For questions about solvers, we HAVE to see the complete output of -ksp_view so we
> know what we are dealing with. Its also nice to have -ksp_monitor_true_residual -ksp_converged_reason
> 
>   Matt
> 
> Yes -- sorry, those are both in inline files and are called in the reverse order that I wrote them out.  
> 
> I've attached the output of 
> 
> $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason > kspview.log
> 
>  
>  
> And then setup the KSP with
>   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
> 
> The matrix QTBNQ does not change, only the rhs of the system is updated.
> 
> We run this with `-pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.  
> 
> Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
> 
>   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
> 
> and the same code diverges after 1 timestep and returns a -8 KSP_DIVERGED_INDEFINITE_PC
> 
> This is weird, especially because if we change nsmooths to 2, it runs for 264 timesteps and the returns the same error.  But we have explicitly set KSPSetReusePreconditioner so it should be using the same PC, right?
> 
> Change nsmooths to 3 and it again diverges after 1 timestep.  
> 
> Change nsmooths to 4 and it runs to completion.  
> 
> It seems like either gamg's behavior has changed, or that KSPSetNullSpace was doing something implicitly that we now need to do explicitly in addition to MatSetNullSpace?  
> 
> Thanks, 
> Gil Forsyth
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> <kspview.log>


From gforsyth at gwu.edu  Tue Sep 29 12:00:59 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Tue, 29 Sep 2015 13:00:59 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
Message-ID: <CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>

Hi Barry,

We aren't explicitly setting GMRES anywhere in the code and I'm not sure
why it's being used.  Running our 3.5.4 code using KSPSetNullSpace works
with:

$PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
-poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
-poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
kspview3.5.4

and shows that the coarse grid solver is of type:preonly

running the newer version that uses MatSetNullSpace in its stead and adding
in -poisson_mg_coarse_ksp_type preonly

$PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
-poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
-poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
-poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
kspview3.6.1

still shows

KSP Object:(poisson_) 1 MPI processes
  type: cg
  maximum iterations=10000
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
  left preconditioning
  using nonzero initial guess
  using PRECONDITIONED norm type for convergence test
PC Object:(poisson_) 1 MPI processes
  type: gamg
    MG: type is MULTIPLICATIVE, levels=3 cycles=v
      Cycles per PCApply=1
      Using Galerkin computed coarse grid matrices
      GAMG specific options
        Threshold for dropping small values from graph 0
        AGG specific options
          Symmetric graph false
  Coarse grid solver -- level -------------------------------
    KSP Object:    (poisson_mg_coarse_)     1 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test


both logs are attached.


On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    This can't work. You can't use a GMRES inside a CG.   Try changing to
> -poisson_mg_coarse_ksp_type preonly
>
> KSP Object:(poisson_) 1 MPI processes
>   type: cg
>
> KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>       type: gmres
>         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>         GMRES: happy breakdown tolerance 1e-30
>       maximum iterations=1, initial guess is zero
>
>
> > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> >
> >
> > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > Hi all,
> >
> > I've been having some trouble with what should be a relatively simple
> update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
> >
> > I'm getting indefinite PC errors for a simple lid-driven cavity test
> problem, 32x32 at Re 100
> >
> > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set the
> null space.  This is for a 2D Poisson system with no immersed boundary and
> so the null space is the constant vector.
> >
> >   MatNullSpace nsp;
> >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
> &nsp); CHKERRQ(ierr);
> >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
> >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
> >
> > Clearly this has to happen in the reverse order, since ksp2 would not be
> created yet.
> >
> > For questions about solvers, we HAVE to see the complete output of
> -ksp_view so we
> > know what we are dealing with. Its also nice to have
> -ksp_monitor_true_residual -ksp_converged_reason
> >
> >   Matt
> >
> > Yes -- sorry, those are both in inline files and are called in the
> reverse order that I wrote them out.
> >
> > I've attached the output of
> >
> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
> kspview.log
> >
> >
> >
> > And then setup the KSP with
> >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
> >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
> >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
> >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
> >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
> >
> > The matrix QTBNQ does not change, only the rhs of the system is updated.
> >
> > We run this with `-pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths
> 1` and everything seems to work as expected.
> >
> > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
> >
> >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
> >
> > and the same code diverges after 1 timestep and returns a -8
> KSP_DIVERGED_INDEFINITE_PC
> >
> > This is weird, especially because if we change nsmooths to 2, it runs
> for 264 timesteps and the returns the same error.  But we have explicitly
> set KSPSetReusePreconditioner so it should be using the same PC, right?
> >
> > Change nsmooths to 3 and it again diverges after 1 timestep.
> >
> > Change nsmooths to 4 and it runs to completion.
> >
> > It seems like either gamg's behavior has changed, or that
> KSPSetNullSpace was doing something implicitly that we now need to do
> explicitly in addition to MatSetNullSpace?
> >
> > Thanks,
> > Gil Forsyth
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > <kspview.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/c175640b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kspview3.5.4
Type: application/octet-stream
Size: 7804 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/c175640b/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kspview3.6.1
Type: application/octet-stream
Size: 10343 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/c175640b/attachment-0003.obj>

From bsmith at mcs.anl.gov  Tue Sep 29 12:04:02 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 29 Sep 2015 12:04:02 -0500
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
Message-ID: <F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>


  Update your PETSc


> On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> 
> Hi Barry, 
> 
> We aren't explicitly setting GMRES anywhere in the code and I'm not sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace works with:
> 
> $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason > kspview3.5.4
> 
> and shows that the coarse grid solver is of type:preonly
> 
> running the newer version that uses MatSetNullSpace in its stead and adding in -poisson_mg_coarse_ksp_type preonly
> 
> $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason > kspview3.6.1
> 
> still shows
> 
> KSP Object:(poisson_) 1 MPI processes
>   type: cg
>   maximum iterations=10000
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>   left preconditioning
>   using nonzero initial guess
>   using PRECONDITIONED norm type for convergence test
> PC Object:(poisson_) 1 MPI processes
>   type: gamg
>     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>       Cycles per PCApply=1
>       Using Galerkin computed coarse grid matrices
>       GAMG specific options
>         Threshold for dropping small values from graph 0
>         AGG specific options
>           Symmetric graph false
>   Coarse grid solver -- level -------------------------------
>     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>       type: gmres
>         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>         GMRES: happy breakdown tolerance 1e-30
>       maximum iterations=1, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>       left preconditioning
>       using NONE norm type for convergence test
> 
> 
> both logs are attached.
> 
> 
> On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>    This can't work. You can't use a GMRES inside a CG.   Try changing to -poisson_mg_coarse_ksp_type preonly
> 
> KSP Object:(poisson_) 1 MPI processes
>   type: cg
> 
> KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>       type: gmres
>         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>         GMRES: happy breakdown tolerance 1e-30
>       maximum iterations=1, initial guess is zero
> 
> 
> > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> >
> >
> > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > Hi all,
> >
> > I've been having some trouble with what should be a relatively simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
> >
> > I'm getting indefinite PC errors for a simple lid-driven cavity test problem, 32x32 at Re 100
> >
> > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set the null space.  This is for a 2D Poisson system with no immersed boundary and so the null space is the constant vector.
> >
> >   MatNullSpace nsp;
> >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL, &nsp); CHKERRQ(ierr);
> >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
> >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
> >
> > Clearly this has to happen in the reverse order, since ksp2 would not be created yet.
> >
> > For questions about solvers, we HAVE to see the complete output of -ksp_view so we
> > know what we are dealing with. Its also nice to have -ksp_monitor_true_residual -ksp_converged_reason
> >
> >   Matt
> >
> > Yes -- sorry, those are both in inline files and are called in the reverse order that I wrote them out.
> >
> > I've attached the output of
> >
> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason > kspview.log
> >
> >
> >
> > And then setup the KSP with
> >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
> >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
> >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
> >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
> >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
> >
> > The matrix QTBNQ does not change, only the rhs of the system is updated.
> >
> > We run this with `-pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
> >
> > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
> >
> >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
> >
> > and the same code diverges after 1 timestep and returns a -8 KSP_DIVERGED_INDEFINITE_PC
> >
> > This is weird, especially because if we change nsmooths to 2, it runs for 264 timesteps and the returns the same error.  But we have explicitly set KSPSetReusePreconditioner so it should be using the same PC, right?
> >
> > Change nsmooths to 3 and it again diverges after 1 timestep.
> >
> > Change nsmooths to 4 and it runs to completion.
> >
> > It seems like either gamg's behavior has changed, or that KSPSetNullSpace was doing something implicitly that we now need to do explicitly in addition to MatSetNullSpace?
> >
> > Thanks,
> > Gil Forsyth
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> >
> > <kspview.log>
> 
> 
> <kspview3.5.4><kspview3.6.1>


From gforsyth at gwu.edu  Tue Sep 29 12:08:26 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Tue, 29 Sep 2015 13:08:26 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
Message-ID: <CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>

PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that the
behavior seems to have changed between versions.  The only difference in
our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
MatSetNullSpace.

On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Update your PETSc
>
>
> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> >
> > Hi Barry,
> >
> > We aren't explicitly setting GMRES anywhere in the code and I'm not sure
> why it's being used.  Running our 3.5.4 code using KSPSetNullSpace works
> with:
> >
> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
> kspview3.5.4
> >
> > and shows that the coarse grid solver is of type:preonly
> >
> > running the newer version that uses MatSetNullSpace in its stead and
> adding in -poisson_mg_coarse_ksp_type preonly
> >
> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
> kspview3.6.1
> >
> > still shows
> >
> > KSP Object:(poisson_) 1 MPI processes
> >   type: cg
> >   maximum iterations=10000
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >   left preconditioning
> >   using nonzero initial guess
> >   using PRECONDITIONED norm type for convergence test
> > PC Object:(poisson_) 1 MPI processes
> >   type: gamg
> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
> >       Cycles per PCApply=1
> >       Using Galerkin computed coarse grid matrices
> >       GAMG specific options
> >         Threshold for dropping small values from graph 0
> >         AGG specific options
> >           Symmetric graph false
> >   Coarse grid solver -- level -------------------------------
> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
> >       type: gmres
> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> >         GMRES: happy breakdown tolerance 1e-30
> >       maximum iterations=1, initial guess is zero
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >       left preconditioning
> >       using NONE norm type for convergence test
> >
> >
> > both logs are attached.
> >
> >
> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >
> >    This can't work. You can't use a GMRES inside a CG.   Try changing to
> -poisson_mg_coarse_ksp_type preonly
> >
> > KSP Object:(poisson_) 1 MPI processes
> >   type: cg
> >
> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
> >       type: gmres
> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> >         GMRES: happy breakdown tolerance 1e-30
> >       maximum iterations=1, initial guess is zero
> >
> >
> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > >
> > >
> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
> wrote:
> > > Hi all,
> > >
> > > I've been having some trouble with what should be a relatively simple
> update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
> > >
> > > I'm getting indefinite PC errors for a simple lid-driven cavity test
> problem, 32x32 at Re 100
> > >
> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set
> the null space.  This is for a 2D Poisson system with no immersed boundary
> and so the null space is the constant vector.
> > >
> > >   MatNullSpace nsp;
> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
> &nsp); CHKERRQ(ierr);
> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
> > >
> > > Clearly this has to happen in the reverse order, since ksp2 would not
> be created yet.
> > >
> > > For questions about solvers, we HAVE to see the complete output of
> -ksp_view so we
> > > know what we are dealing with. Its also nice to have
> -ksp_monitor_true_residual -ksp_converged_reason
> > >
> > >   Matt
> > >
> > > Yes -- sorry, those are both in inline files and are called in the
> reverse order that I wrote them out.
> > >
> > > I've attached the output of
> > >
> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
> kspview.log
> > >
> > >
> > >
> > > And then setup the KSP with
> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
> > >
> > > The matrix QTBNQ does not change, only the rhs of the system is
> updated.
> > >
> > > We run this with `-pc_type gamg -pc_gamg_type agg
> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
> > >
> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
> > >
> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
> > >
> > > and the same code diverges after 1 timestep and returns a -8
> KSP_DIVERGED_INDEFINITE_PC
> > >
> > > This is weird, especially because if we change nsmooths to 2, it runs
> for 264 timesteps and the returns the same error.  But we have explicitly
> set KSPSetReusePreconditioner so it should be using the same PC, right?
> > >
> > > Change nsmooths to 3 and it again diverges after 1 timestep.
> > >
> > > Change nsmooths to 4 and it runs to completion.
> > >
> > > It seems like either gamg's behavior has changed, or that
> KSPSetNullSpace was doing something implicitly that we now need to do
> explicitly in addition to MatSetNullSpace?
> > >
> > > Thanks,
> > > Gil Forsyth
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > -- Norbert Wiener
> > >
> > > <kspview.log>
> >
> >
> > <kspview3.5.4><kspview3.6.1>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/ab924a73/attachment.html>

From knepley at gmail.com  Tue Sep 29 12:10:49 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 29 Sep 2015 12:10:49 -0500
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
Message-ID: <CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>

On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:

> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that
> the behavior seems to have changed between versions.  The only difference
> in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
> MatSetNullSpace.
>

Mark made some GAMG changes which were later reversed because they had
unintended consequences like this.
I think what Barry means is, "you should get the behavior you expect using
the master branch from PETSc development"

  Thanks,

     Matt


> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   Update your PETSc
>>
>>
>> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> >
>> > Hi Barry,
>> >
>> > We aren't explicitly setting GMRES anywhere in the code and I'm not
>> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>> works with:
>> >
>> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>> kspview3.5.4
>> >
>> > and shows that the coarse grid solver is of type:preonly
>> >
>> > running the newer version that uses MatSetNullSpace in its stead and
>> adding in -poisson_mg_coarse_ksp_type preonly
>> >
>> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>> kspview3.6.1
>> >
>> > still shows
>> >
>> > KSP Object:(poisson_) 1 MPI processes
>> >   type: cg
>> >   maximum iterations=10000
>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> >   left preconditioning
>> >   using nonzero initial guess
>> >   using PRECONDITIONED norm type for convergence test
>> > PC Object:(poisson_) 1 MPI processes
>> >   type: gamg
>> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>> >       Cycles per PCApply=1
>> >       Using Galerkin computed coarse grid matrices
>> >       GAMG specific options
>> >         Threshold for dropping small values from graph 0
>> >         AGG specific options
>> >           Symmetric graph false
>> >   Coarse grid solver -- level -------------------------------
>> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>> >       type: gmres
>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> >         GMRES: happy breakdown tolerance 1e-30
>> >       maximum iterations=1, initial guess is zero
>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> >       left preconditioning
>> >       using NONE norm type for convergence test
>> >
>> >
>> > both logs are attached.
>> >
>> >
>> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> >
>> >    This can't work. You can't use a GMRES inside a CG.   Try changing
>> to -poisson_mg_coarse_ksp_type preonly
>> >
>> > KSP Object:(poisson_) 1 MPI processes
>> >   type: cg
>> >
>> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>> >       type: gmres
>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> >         GMRES: happy breakdown tolerance 1e-30
>> >       maximum iterations=1, initial guess is zero
>> >
>> >
>> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > >
>> > >
>> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>> wrote:
>> > > Hi all,
>> > >
>> > > I've been having some trouble with what should be a relatively simple
>> update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>> > >
>> > > I'm getting indefinite PC errors for a simple lid-driven cavity test
>> problem, 32x32 at Re 100
>> > >
>> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set
>> the null space.  This is for a 2D Poisson system with no immersed boundary
>> and so the null space is the constant vector.
>> > >
>> > >   MatNullSpace nsp;
>> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
>> &nsp); CHKERRQ(ierr);
>> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>> > >
>> > > Clearly this has to happen in the reverse order, since ksp2 would not
>> be created yet.
>> > >
>> > > For questions about solvers, we HAVE to see the complete output of
>> -ksp_view so we
>> > > know what we are dealing with. Its also nice to have
>> -ksp_monitor_true_residual -ksp_converged_reason
>> > >
>> > >   Matt
>> > >
>> > > Yes -- sorry, those are both in inline files and are called in the
>> reverse order that I wrote them out.
>> > >
>> > > I've attached the output of
>> > >
>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1 -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>> kspview.log
>> > >
>> > >
>> > >
>> > > And then setup the KSP with
>> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>> > >
>> > > The matrix QTBNQ does not change, only the rhs of the system is
>> updated.
>> > >
>> > > We run this with `-pc_type gamg -pc_gamg_type agg
>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>> > >
>> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>> > >
>> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>> > >
>> > > and the same code diverges after 1 timestep and returns a -8
>> KSP_DIVERGED_INDEFINITE_PC
>> > >
>> > > This is weird, especially because if we change nsmooths to 2, it runs
>> for 264 timesteps and the returns the same error.  But we have explicitly
>> set KSPSetReusePreconditioner so it should be using the same PC, right?
>> > >
>> > > Change nsmooths to 3 and it again diverges after 1 timestep.
>> > >
>> > > Change nsmooths to 4 and it runs to completion.
>> > >
>> > > It seems like either gamg's behavior has changed, or that
>> KSPSetNullSpace was doing something implicitly that we now need to do
>> explicitly in addition to MatSetNullSpace?
>> > >
>> > > Thanks,
>> > > Gil Forsyth
>> > >
>> > >
>> > >
>> > > --
>> > > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > > -- Norbert Wiener
>> > >
>> > > <kspview.log>
>> >
>> >
>> > <kspview3.5.4><kspview3.6.1>
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/f70562fb/attachment-0001.html>

From gforsyth at gwu.edu  Tue Sep 29 12:12:44 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Tue, 29 Sep 2015 13:12:44 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
Message-ID: <CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>

Ah, got it.  I'll checkout the master branch and see if the behavior
persists.

Many thanks,
Gil

On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>
>> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that
>> the behavior seems to have changed between versions.  The only difference
>> in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
>> MatSetNullSpace.
>>
>
> Mark made some GAMG changes which were later reversed because they had
> unintended consequences like this.
> I think what Barry means is, "you should get the behavior you expect using
> the master branch from PETSc development"
>
>   Thanks,
>
>      Matt
>
>
>> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>   Update your PETSc
>>>
>>>
>>> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>> >
>>> > Hi Barry,
>>> >
>>> > We aren't explicitly setting GMRES anywhere in the code and I'm not
>>> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>>> works with:
>>> >
>>> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>> kspview3.5.4
>>> >
>>> > and shows that the coarse grid solver is of type:preonly
>>> >
>>> > running the newer version that uses MatSetNullSpace in its stead and
>>> adding in -poisson_mg_coarse_ksp_type preonly
>>> >
>>> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>> kspview3.6.1
>>> >
>>> > still shows
>>> >
>>> > KSP Object:(poisson_) 1 MPI processes
>>> >   type: cg
>>> >   maximum iterations=10000
>>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>> >   left preconditioning
>>> >   using nonzero initial guess
>>> >   using PRECONDITIONED norm type for convergence test
>>> > PC Object:(poisson_) 1 MPI processes
>>> >   type: gamg
>>> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>> >       Cycles per PCApply=1
>>> >       Using Galerkin computed coarse grid matrices
>>> >       GAMG specific options
>>> >         Threshold for dropping small values from graph 0
>>> >         AGG specific options
>>> >           Symmetric graph false
>>> >   Coarse grid solver -- level -------------------------------
>>> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>> >       type: gmres
>>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>> >         GMRES: happy breakdown tolerance 1e-30
>>> >       maximum iterations=1, initial guess is zero
>>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>> >       left preconditioning
>>> >       using NONE norm type for convergence test
>>> >
>>> >
>>> > both logs are attached.
>>> >
>>> >
>>> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> >
>>> >    This can't work. You can't use a GMRES inside a CG.   Try changing
>>> to -poisson_mg_coarse_ksp_type preonly
>>> >
>>> > KSP Object:(poisson_) 1 MPI processes
>>> >   type: cg
>>> >
>>> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>> >       type: gmres
>>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>> >         GMRES: happy breakdown tolerance 1e-30
>>> >       maximum iterations=1, initial guess is zero
>>> >
>>> >
>>> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>> > >
>>> > >
>>> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>>> wrote:
>>> > > Hi all,
>>> > >
>>> > > I've been having some trouble with what should be a relatively
>>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>> > >
>>> > > I'm getting indefinite PC errors for a simple lid-driven cavity test
>>> problem, 32x32 at Re 100
>>> > >
>>> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set
>>> the null space.  This is for a 2D Poisson system with no immersed boundary
>>> and so the null space is the constant vector.
>>> > >
>>> > >   MatNullSpace nsp;
>>> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
>>> &nsp); CHKERRQ(ierr);
>>> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>> > >
>>> > > Clearly this has to happen in the reverse order, since ksp2 would
>>> not be created yet.
>>> > >
>>> > > For questions about solvers, we HAVE to see the complete output of
>>> -ksp_view so we
>>> > > know what we are dealing with. Its also nice to have
>>> -ksp_monitor_true_residual -ksp_converged_reason
>>> > >
>>> > >   Matt
>>> > >
>>> > > Yes -- sorry, those are both in inline files and are called in the
>>> reverse order that I wrote them out.
>>> > >
>>> > > I've attached the output of
>>> > >
>>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>> -poisson_ksp_converged_reason > kspview.log
>>> > >
>>> > >
>>> > >
>>> > > And then setup the KSP with
>>> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>>> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>>> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>> > >
>>> > > The matrix QTBNQ does not change, only the rhs of the system is
>>> updated.
>>> > >
>>> > > We run this with `-pc_type gamg -pc_gamg_type agg
>>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>>> > >
>>> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>> > >
>>> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>> > >
>>> > > and the same code diverges after 1 timestep and returns a -8
>>> KSP_DIVERGED_INDEFINITE_PC
>>> > >
>>> > > This is weird, especially because if we change nsmooths to 2, it
>>> runs for 264 timesteps and the returns the same error.  But we have
>>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>>> right?
>>> > >
>>> > > Change nsmooths to 3 and it again diverges after 1 timestep.
>>> > >
>>> > > Change nsmooths to 4 and it runs to completion.
>>> > >
>>> > > It seems like either gamg's behavior has changed, or that
>>> KSPSetNullSpace was doing something implicitly that we now need to do
>>> explicitly in addition to MatSetNullSpace?
>>> > >
>>> > > Thanks,
>>> > > Gil Forsyth
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > > -- Norbert Wiener
>>> > >
>>> > > <kspview.log>
>>> >
>>> >
>>> > <kspview3.5.4><kspview3.6.1>
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/58ba6579/attachment.html>

From balay at mcs.anl.gov  Tue Sep 29 12:21:26 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 29 Sep 2015 12:21:26 -0500
Subject: [petsc-users] Error when compiling petsc on windows
In-Reply-To: <CAEpUo3gfkkRLcNya8pxck4gNyVWHLtFgnDSGrFKbmtcixbE9pQ@mail.gmail.com>
References: <CAEpUo3gfkkRLcNya8pxck4gNyVWHLtFgnDSGrFKbmtcixbE9pQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.20.1509291219270.28875@asterix>

If you compile an example using a makefile - you should see something like:

>>>>>>>>
sbalay at ps3 ~/petsc.clone/src/ksp/ksp/examples/tutorials
$ make ex2
/home/sbalay/petsc.clone/bin/win32fe/win32fe cl -o ex2.o -c -MT -Z7   -I/home/sbalay/petsc.clone/include -I/home/sbalay/petsc.clone/arch-mswin/include -I/cygdrive/c/Program\ Files/MPICH2/include    `pwd`/ex2.c
ex2.c
/home/sbalay/petsc.clone/bin/win32fe/win32fe cl -MT -Z7  -o ex2 ex2.o  -L/home/sbalay/petsc.clone/arch-mswin/lib  -lpetsc /cygdrive/c/Program\ Files/Intel/MKL/ia32/lib/mkl_c_dll.lib /cygdrive/c/Program\ Files/MPICH2/lib/fmpich2.lib /cygdrive/c/Program\ Files/MPICH2/lib/fmpich2g.lib /cygdrive/c/Program\ Files/MPICH2/lib/fmpich2s.lib /cygdrive/c/Program\ Files/MPICH2/lib/mpi.lib Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib  
/usr/bin/rm -f ex2.o

<<<<<<<<

>From this list - there are some comiler libraries 'Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib' listed
in the link linke.

One of them should provide __imp_GetUserNameA. [Perhaps you need to add them all to the link linke]

Satish

On Tue, 29 Sep 2015, Bruno Ramon Batista Fernande wrote:

> Dear petsc users,
> 
> I have been trying to use use PETSC on my project on windows. I have
> followed all steps and compiled petsc on windows using cygwin and cmake.
> Then I included it on my visual studio project with all other dependencies.
> After fixing several errors I got stuck with this one:
> 
> Error 1 error LNK2019: unresolved external symbol __imp_GetUserNameA
> referenced in function PetscGetUserName libpetsc.lib(fuser.o)
> 
> Anyone has any Idea of which library I should include to solve such
> problem?
> 
> I am using windows 8 machine with visual studio 2013 and Intel fortran
> 2013, MS MPI, metis, parmetis, and mkl. All libraries are compiled to x64
> and the platform on the VS project was set to x64. I have also activated
> the use of common windows libraries on the VS project to solve other
> problems. This is the configure I used:
> 
> ./configure --with-cc='win32fe cl' --with-fc='win32fe ifort'
> --prefix=/cygdrive/c/Installed/petsc-opt/ --PETSC_ARCH=windows-intel-opt
> --with-parmetis=1
> --with-parmetis-include=/cygdrive/c/Installed/Parmetis/include_all/
> --with-parmetis-lib=\[/cygdrive/c/Installed/Parmetis/lib/parmetis.lib\]
> --with-metis=1
> --with-metis-include=/cygdrive/c/Installed/Parmetis/include_all/
> --with-metis-lib=\[/cygdrive/c/Installed/Parmetis/lib/metis.lib\]
> --with-mpi-include=[/cygdrive/c/Installed/MPI/Include/,/cygdrive/c/Installed/MPI/Include/x64/]
> --with-mpi-lib=[/cygdrive/c/Installed/MPI/Lib/x64/msmpi.lib,/cygdrive/c/Installed/MPI/Lib/x64/msmpifec.lib]
> --with-blas-lapack-lib=[/cygdrive/c/Installed/mkl/lib/intel64/mkl_intel_lp64.lib,/cygdrive/c/Installed/mkl/lib/intel64/mkl_core.lib,/cygdrive/c/Installed/mkl/lib/intel64/mkl_sequential.lib]
> --with-scalapack-include=/cygdrive/c/Installed/mkl/include
> --with-scalapack-lib=[/cygdrive/c/Installed/mkl/lib/intel64/mkl_scalapack_lp64.lib,/cygdrive/c/Installed/mkl/lib/intel64/mkl_blacs_msmpi_lp64.lib]
> --with-fortran-interfaces=1 --with-debugging=0
> 
> regards,
> 


From brbfernandes at utexas.edu  Tue Sep 29 15:30:34 2015
From: brbfernandes at utexas.edu (Bruno Ramon Batista Fernande)
Date: Tue, 29 Sep 2015 15:30:34 -0500
Subject: [petsc-users] Error when compiling petsc on windows
In-Reply-To: <alpine.LFD.2.20.1509291219270.28875@asterix>
References: <CAEpUo3gfkkRLcNya8pxck4gNyVWHLtFgnDSGrFKbmtcixbE9pQ@mail.gmail.com>
	<alpine.LFD.2.20.1509291219270.28875@asterix>
Message-ID: <CAEpUo3jzjm+bUOcFf7a9Enx4YbCA8WawBw-OG4daRwnBt-FHkw@mail.gmail.com>

Hi Satish,

It worked with the mentioned libraries. Thank you very much.

Regards,

Bruno Fernandes

On Tue, Sep 29, 2015 at 12:21 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> If you compile an example using a makefile - you should see something like:
>
> >>>>>>>>
> sbalay at ps3 ~/petsc.clone/src/ksp/ksp/examples/tutorials
> $ make ex2
> /home/sbalay/petsc.clone/bin/win32fe/win32fe cl -o ex2.o -c -MT -Z7
>  -I/home/sbalay/petsc.clone/include
> -I/home/sbalay/petsc.clone/arch-mswin/include -I/cygdrive/c/Program\
> Files/MPICH2/include    `pwd`/ex2.c
> ex2.c
> /home/sbalay/petsc.clone/bin/win32fe/win32fe cl -MT -Z7  -o ex2 ex2.o
> -L/home/sbalay/petsc.clone/arch-mswin/lib  -lpetsc /cygdrive/c/Program\
> Files/Intel/MKL/ia32/lib/mkl_c_dll.lib /cygdrive/c/Program\
> Files/MPICH2/lib/fmpich2.lib /cygdrive/c/Program\
> Files/MPICH2/lib/fmpich2g.lib /cygdrive/c/Program\
> Files/MPICH2/lib/fmpich2s.lib /cygdrive/c/Program\ Files/MPICH2/lib/mpi.lib
> Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib
> /usr/bin/rm -f ex2.o
>
> <<<<<<<<
>
> From this list - there are some comiler libraries 'Gdi32.lib User32.lib
> Advapi32.lib Kernel32.lib Ws2_32.lib' listed
> in the link linke.
>
> One of them should provide __imp_GetUserNameA. [Perhaps you need to add
> them all to the link linke]
>
> Satish
>
> On Tue, 29 Sep 2015, Bruno Ramon Batista Fernande wrote:
>
> > Dear petsc users,
> >
> > I have been trying to use use PETSC on my project on windows. I have
> > followed all steps and compiled petsc on windows using cygwin and cmake.
> > Then I included it on my visual studio project with all other
> dependencies.
> > After fixing several errors I got stuck with this one:
> >
> > Error 1 error LNK2019: unresolved external symbol __imp_GetUserNameA
> > referenced in function PetscGetUserName libpetsc.lib(fuser.o)
> >
> > Anyone has any Idea of which library I should include to solve such
> > problem?
> >
> > I am using windows 8 machine with visual studio 2013 and Intel fortran
> > 2013, MS MPI, metis, parmetis, and mkl. All libraries are compiled to x64
> > and the platform on the VS project was set to x64. I have also activated
> > the use of common windows libraries on the VS project to solve other
> > problems. This is the configure I used:
> >
> > ./configure --with-cc='win32fe cl' --with-fc='win32fe ifort'
> > --prefix=/cygdrive/c/Installed/petsc-opt/ --PETSC_ARCH=windows-intel-opt
> > --with-parmetis=1
> > --with-parmetis-include=/cygdrive/c/Installed/Parmetis/include_all/
> > --with-parmetis-lib=\[/cygdrive/c/Installed/Parmetis/lib/parmetis.lib\]
> > --with-metis=1
> > --with-metis-include=/cygdrive/c/Installed/Parmetis/include_all/
> > --with-metis-lib=\[/cygdrive/c/Installed/Parmetis/lib/metis.lib\]
> >
> --with-mpi-include=[/cygdrive/c/Installed/MPI/Include/,/cygdrive/c/Installed/MPI/Include/x64/]
> >
> --with-mpi-lib=[/cygdrive/c/Installed/MPI/Lib/x64/msmpi.lib,/cygdrive/c/Installed/MPI/Lib/x64/msmpifec.lib]
> >
> --with-blas-lapack-lib=[/cygdrive/c/Installed/mkl/lib/intel64/mkl_intel_lp64.lib,/cygdrive/c/Installed/mkl/lib/intel64/mkl_core.lib,/cygdrive/c/Installed/mkl/lib/intel64/mkl_sequential.lib]
> > --with-scalapack-include=/cygdrive/c/Installed/mkl/include
> >
> --with-scalapack-lib=[/cygdrive/c/Installed/mkl/lib/intel64/mkl_scalapack_lp64.lib,/cygdrive/c/Installed/mkl/lib/intel64/mkl_blacs_msmpi_lp64.lib]
> > --with-fortran-interfaces=1 --with-debugging=0
> >
> > regards,
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/f2eb82d2/attachment-0001.html>

From E.Mueller at bath.ac.uk  Wed Sep 30 03:00:50 2015
From: E.Mueller at bath.ac.uk (Eike Mueller)
Date: Wed, 30 Sep 2015 08:00:50 +0000
Subject: [petsc-users] fieldsplit-preconditioner for block-system of linear
	equations
Message-ID: <FF7FA0DF-D0EB-4DF0-B6E1-34687F8C1976@bath.ac.uk>

Dear PETSc,

I am solving a linear system of equations

(1) A.X = B

where the vector X is divided into chunks, such that X= (x_1,x_2,?,x_k) and each of the vectors x_i has length n_i (same for the vector B=(b_1,b_2,?,b_k)). k=5 in my case, and n_i >> 1. This partitioning implies that the matrix has a block-structure, where the submatrices A_{ij} are of size n_i x n_j. So explicitly for k=3:

        A_{1,1}.x_1 + A_{1,2}.x_2 + A_{1,3}.x_3 = b_1
(2)    A_{2,1}.x_1 + A_{2,2}.x_2 + A_{2,3}.x_3 = b_2
        A_{3,1}.x_1 + A_{3,2}.x_2 + A_{3,3}.x_3 = b_3

I now want to solve this system with a Krylov-method (say, GMRES) and precondition it with a field-split preconditioner, such that the Schur-complement is formed in the 1-block. I know how to do this if I have assembled the big matrix A, and I store all x_i in one big vector X (I construct the index-sets corresponding to the vectors x_1 and (x_2,x_3), and then call PCFieldSplitSetIS()). This all works without any problems.

However, I now want to do this for an existing code which

1. Assembles the matrices A_{ij} separately (i.e. they are not stored as part of a big matrix A, but are stored in independent memory locations)
2. Stores the vectors x_i separately, i.e. they are not stored as parts of one big chunk of memory of size n_1+?+n_k

I can of course implement the matrix application via a matrix shell, but is there still an easy way of using the fieldsplit preconditioner?

The naive way I can think of is to allocate a new big matrix A, and copy the A_{ij} into the corresponding blocks. Then allocate big vectors x and b, and also copy in/out the data before and after the solve. This, however, seems to be wasteful, so before I go down this route I wanted to double check if there is a way around it, since this seems to be a very common problem?

Thanks a lot,

Eike


From no-reply at dropboxmail.com  Wed Sep 30 09:17:58 2015
From: no-reply at dropboxmail.com (Dropbox)
Date: Wed, 30 Sep 2015 14:17:58 +0000
Subject: [petsc-users] =?utf-8?q?Manuel_te_invit=C3=B3_a_unirte_a_la_Copa_?=
 =?utf-8?q?Universitaria_de_Dropbox=2E?=
Message-ID: <000001501e9dede2-69818714-ed33-4c2a-a3eb-41556d2c721b-000000@us-west-2.amazonses.com>

Hola, Petsc::

Manuel te invit? a unirte a la Copa universitaria de Dropbox. La Copa universitaria te ofrece la oportunidad de trabajar en equipo con otros alumnos y de obtener hasta 25?GB de espacio gratuito en Dropbox durante 2?a?os.

Para ganar m?s espacio, invita amigos y usa Dropbox para compartir archivos, colaborar y hacer tu trabajo.

??nete a la Copa Universitaria![2]

Ten en cuenta que, para poder participar, debes ser un alumno inscripto en una instituci?n de educaci?n superior. M?s informaci?n acerca de la Copa universitaria[3]

?Divi?rtete con Dropbox!


 
 Trabaja en conjunto para que todos ganen.

____________________________________________________
Si prefieres no recibir invitaciones de Dropbox, accede aqu?[4].
Dropbox, Inc., PO Box 77767, San Francisco, CA 94107

[1]: https://www.dropbox.com/l/QrJNg9qPbebdLpEjQJ3D9w/campuscup?text=1
[2]: https://www.dropbox.com/l/uD8C3rbxD0zFpstBjjZuzg/campuscup?text=1
[3]: https://www.dropbox.com/l/wer7RvzLFZhr5WyXzw1zmo/help/9131?text=1
[4]: https://www.dropbox.com/l/d7TPgGqKkeYYqTJF6X5HWD?text=1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150930/2862f6b6/attachment.html>

From knepley at gmail.com  Wed Sep 30 09:27:10 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Sep 2015 09:27:10 -0500
Subject: [petsc-users] fieldsplit-preconditioner for block-system of
 linear equations
In-Reply-To: <FF7FA0DF-D0EB-4DF0-B6E1-34687F8C1976@bath.ac.uk>
References: <FF7FA0DF-D0EB-4DF0-B6E1-34687F8C1976@bath.ac.uk>
Message-ID: <CAMYG4GnSp__-G_g79S9rGYFaSoUUdYV9UC1R-qRt6OjW3Pw8Ow@mail.gmail.com>

On Wed, Sep 30, 2015 at 3:00 AM, Eike Mueller <E.Mueller at bath.ac.uk> wrote:

> Dear PETSc,
>
> I am solving a linear system of equations
>
> (1) A.X = B
>
> where the vector X is divided into chunks, such that X= (x_1,x_2,?,x_k)
> and each of the vectors x_i has length n_i (same for the vector
> B=(b_1,b_2,?,b_k)). k=5 in my case, and n_i >> 1. This partitioning implies
> that the matrix has a block-structure, where the submatrices A_{ij} are of
> size n_i x n_j. So explicitly for k=3:
>
>         A_{1,1}.x_1 + A_{1,2}.x_2 + A_{1,3}.x_3 = b_1
> (2)    A_{2,1}.x_1 + A_{2,2}.x_2 + A_{2,3}.x_3 = b_2
>         A_{3,1}.x_1 + A_{3,2}.x_2 + A_{3,3}.x_3 = b_3
>
> I now want to solve this system with a Krylov-method (say, GMRES) and
> precondition it with a field-split preconditioner, such that the
> Schur-complement is formed in the 1-block. I know how to do this if I have
> assembled the big matrix A, and I store all x_i in one big vector X (I
> construct the index-sets corresponding to the vectors x_1 and (x_2,x_3),
> and then call PCFieldSplitSetIS()). This all works without any problems.
>
> However, I now want to do this for an existing code which
>
> 1. Assembles the matrices A_{ij} separately (i.e. they are not stored as
> part of a big matrix A, but are stored in independent memory locations)
> 2. Stores the vectors x_i separately, i.e. they are not stored as parts of
> one big chunk of memory of size n_1+?+n_k
>
> I can of course implement the matrix application via a matrix shell, but
> is there still an easy way of using the fieldsplit preconditioner?
>

>From your description, it sounds like you could use


http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html

to assemble your separate matrices directly into a large matrix.
Furthermore, if it really benefits you
to have separate memory, you can change the matrix type from MATIJ to
MATNEST dynamically,
but none of your code would change. This would let you check things with
LU, but still optimize when
necessary.

  Thanks,

      Matt


> The naive way I can think of is to allocate a new big matrix A, and copy
> the A_{ij} into the corresponding blocks. Then allocate big vectors x and
> b, and also copy in/out the data before and after the solve. This, however,
> seems to be wasteful, so before I go down this route I wanted to double
> check if there is a way around it, since this seems to be a very common
> problem?
>
> Thanks a lot,
>
> Eike
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150930/6f54b91a/attachment.html>

From matt.landreman at gmail.com  Wed Sep 30 13:06:25 2015
From: matt.landreman at gmail.com (Matt Landreman)
Date: Wed, 30 Sep 2015 14:06:25 -0400
Subject: [petsc-users] Automatically re-solving after MUMPS error
In-Reply-To: <CAJCWK9BihEkqFefFMPjQNBj5gzp6Zfh-r_gdh=Mh+SL5GJcD5A@mail.gmail.com>
References: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
	<CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>
	<CAJCWK9BihEkqFefFMPjQNBj5gzp6Zfh-r_gdh=Mh+SL5GJcD5A@mail.gmail.com>
Message-ID: <CADZ9mvkNH3CvwZigy5cccX5q8oPeHOvDVxPn52q3J3wvn00xWQ@mail.gmail.com>

PETSc developers,

I tried implementing a system for automatically increasing MUMPS ICNTL(14),
along the lines described in this recent thread. If SNESSolve returns ierr
.ne. 0 due to MUMPS error -9, I call SNESDestroy, re-initialize SNES, call
MatMumpsSetIcntl with a larger value of ICNTL(14), call SNESSolve again,
and repeat as needed. The procedure works, but the peak memory required (as
measured by the HPC system) is 50%-100% higher if the MUMPS solve has to be
repeated compared to when MUMPS works on the 1st try (by starting with a
large ICNTL(14)), even though SNESDestroy is called in between the
attempts. Are there some PETSc or MUMPS structures which would not be
deallocated immediately by SNESDestroy?  If so, how do I deallocate them?

Thanks,
Matt Landreman


On Tue, Sep 15, 2015 at 7:47 AM, David Knezevic <david.knezevic at akselos.com>
wrote:

> On Tue, Sep 15, 2015 at 7:29 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Tue, Sep 15, 2015 at 4:30 AM, David Knezevic <
>> david.knezevic at akselos.com> wrote:
>>
>>> In some cases, I get MUMPS error -9, i.e.:
>>> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFO(1)=-9, INFO(2)=98927
>>>
>>> This is easily fixed by re-running the executable with
>>> -mat_mumps_icntl_14 on the commandline.
>>>
>>> However, I would like to update my code in order to do this
>>> automatically, i.e. detect the -9 error and re-run with the appropriate
>>> option. Is there a recommended way to do this? It seems to me that I could
>>> do this with a PETSc error handler (e.g. PetscPushErrorHandler) in order to
>>> call a function that sets the appropriate option and solves again, is that
>>> right? Are there any examples that illustrate this type of thing?
>>>
>>
>> I would not use the error handler. I would just check the ierr return
>> code from the solver. I think you need the
>> INFO output, for which you can use MatMumpsGetInfo().
>>
>
>
> OK, that sounds good (and much simpler than what I had in mind), thanks
> for the help!
>
> David
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150930/8e85f7e1/attachment.html>

From bsmith at mcs.anl.gov  Wed Sep 30 13:16:29 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 30 Sep 2015 13:16:29 -0500
Subject: [petsc-users] Automatically re-solving after MUMPS error
In-Reply-To: <CADZ9mvkNH3CvwZigy5cccX5q8oPeHOvDVxPn52q3J3wvn00xWQ@mail.gmail.com>
References: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
	<CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>
	<CAJCWK9BihEkqFefFMPjQNBj5gzp6Zfh-r_gdh=Mh+SL5GJcD5A@mail.gmail.com>
	<CADZ9mvkNH3CvwZigy5cccX5q8oPeHOvDVxPn52q3J3wvn00xWQ@mail.gmail.com>
Message-ID: <E589862D-F077-4233-A0DD-48F9257F32C3@mcs.anl.gov>


> On Sep 30, 2015, at 1:06 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
> 
> PETSc developers,
> 
> I tried implementing a system for automatically increasing MUMPS ICNTL(14), along the lines described in this recent thread. If SNESSolve returns ierr .ne. 0 due to MUMPS error -9, I call SNESDestroy, re-initialize SNES, call MatMumpsSetIcntl with a larger value of ICNTL(14), call SNESSolve again, and repeat as needed. The procedure works, but the peak memory required (as measured by the HPC system) is 50%-100% higher if the MUMPS solve has to be repeated compared to when MUMPS works on the 1st try (by starting with a large ICNTL(14)), even though SNESDestroy is called in between the attempts. Are there some PETSc or MUMPS structures which would not be deallocated immediately by SNESDestroy?  If so, how do I deallocate them?

   They should be all destroyed automatically for you. You can use PetscMallocDump() after the SNES is destroyed to verify that all that memory is not properly freed. 

   My guess is that your new malloc() with the bigger workspace cannot "reuse" the space that was previously freed; so to the OS it looks like you are using a lot more space but in terms of physical memory you are not using more.

  Barry

> 
> Thanks,
> Matt Landreman
> 
> 
> On Tue, Sep 15, 2015 at 7:47 AM, David Knezevic <david.knezevic at akselos.com> wrote:
> On Tue, Sep 15, 2015 at 7:29 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Sep 15, 2015 at 4:30 AM, David Knezevic <david.knezevic at akselos.com> wrote:
> In some cases, I get MUMPS error -9, i.e.:
> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-9, INFO(2)=98927
> 
> This is easily fixed by re-running the executable with -mat_mumps_icntl_14 on the commandline.
> 
> However, I would like to update my code in order to do this automatically, i.e. detect the -9 error and re-run with the appropriate option. Is there a recommended way to do this? It seems to me that I could do this with a PETSc error handler (e.g. PetscPushErrorHandler) in order to call a function that sets the appropriate option and solves again, is that right? Are there any examples that illustrate this type of thing?
> 
> I would not use the error handler. I would just check the ierr return code from the solver. I think you need the
> INFO output, for which you can use MatMumpsGetInfo().
> 
> 
> OK, that sounds good (and much simpler than what I had in mind), thanks for the help!
> 
> David
> 
> 


From gforsyth at gwu.edu  Wed Sep 30 15:11:14 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Wed, 30 Sep 2015 16:11:14 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
Message-ID: <CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>

Using PETSc master branch solved the problem in serial, but I'm still
seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
runs to completion when I don't use GAMG.  Log is attached for the
following run.

$PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2 $PETIBM_DIR/petibm-git/bin/petibm2d
-directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg
-poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
-poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason


Thanks again,
Gil Forsyth


On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:

> Ah, got it.  I'll checkout the master branch and see if the behavior
> persists.
>
> Many thanks,
> Gil
>
> On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>
>>> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that
>>> the behavior seems to have changed between versions.  The only difference
>>> in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
>>> MatSetNullSpace.
>>>
>>
>> Mark made some GAMG changes which were later reversed because they had
>> unintended consequences like this.
>> I think what Barry means is, "you should get the behavior you expect
>> using the master branch from PETSc development"
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>>
>>>>   Update your PETSc
>>>>
>>>>
>>>> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>> >
>>>> > Hi Barry,
>>>> >
>>>> > We aren't explicitly setting GMRES anywhere in the code and I'm not
>>>> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>>>> works with:
>>>> >
>>>> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
>>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>>> kspview3.5.4
>>>> >
>>>> > and shows that the coarse grid solver is of type:preonly
>>>> >
>>>> > running the newer version that uses MatSetNullSpace in its stead and
>>>> adding in -poisson_mg_coarse_ksp_type preonly
>>>> >
>>>> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
>>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>>> kspview3.6.1
>>>> >
>>>> > still shows
>>>> >
>>>> > KSP Object:(poisson_) 1 MPI processes
>>>> >   type: cg
>>>> >   maximum iterations=10000
>>>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>> >   left preconditioning
>>>> >   using nonzero initial guess
>>>> >   using PRECONDITIONED norm type for convergence test
>>>> > PC Object:(poisson_) 1 MPI processes
>>>> >   type: gamg
>>>> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>>> >       Cycles per PCApply=1
>>>> >       Using Galerkin computed coarse grid matrices
>>>> >       GAMG specific options
>>>> >         Threshold for dropping small values from graph 0
>>>> >         AGG specific options
>>>> >           Symmetric graph false
>>>> >   Coarse grid solver -- level -------------------------------
>>>> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>> >       type: gmres
>>>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>>> Orthogonalization with no iterative refinement
>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>> >       maximum iterations=1, initial guess is zero
>>>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>> >       left preconditioning
>>>> >       using NONE norm type for convergence test
>>>> >
>>>> >
>>>> > both logs are attached.
>>>> >
>>>> >
>>>> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>> >
>>>> >    This can't work. You can't use a GMRES inside a CG.   Try changing
>>>> to -poisson_mg_coarse_ksp_type preonly
>>>> >
>>>> > KSP Object:(poisson_) 1 MPI processes
>>>> >   type: cg
>>>> >
>>>> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>> >       type: gmres
>>>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>>> Orthogonalization with no iterative refinement
>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>> >       maximum iterations=1, initial guess is zero
>>>> >
>>>> >
>>>> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>> > >
>>>> > >
>>>> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <
>>>> knepley at gmail.com> wrote:
>>>> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>> wrote:
>>>> > > Hi all,
>>>> > >
>>>> > > I've been having some trouble with what should be a relatively
>>>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>>> > >
>>>> > > I'm getting indefinite PC errors for a simple lid-driven cavity
>>>> test problem, 32x32 at Re 100
>>>> > >
>>>> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to
>>>> set the null space.  This is for a 2D Poisson system with no immersed
>>>> boundary and so the null space is the constant vector.
>>>> > >
>>>> > >   MatNullSpace nsp;
>>>> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
>>>> &nsp); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>>> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>>> > >
>>>> > > Clearly this has to happen in the reverse order, since ksp2 would
>>>> not be created yet.
>>>> > >
>>>> > > For questions about solvers, we HAVE to see the complete output of
>>>> -ksp_view so we
>>>> > > know what we are dealing with. Its also nice to have
>>>> -ksp_monitor_true_residual -ksp_converged_reason
>>>> > >
>>>> > >   Matt
>>>> > >
>>>> > > Yes -- sorry, those are both in inline files and are called in the
>>>> reverse order that I wrote them out.
>>>> > >
>>>> > > I've attached the output of
>>>> > >
>>>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>>> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
>>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>> -poisson_ksp_converged_reason > kspview.log
>>>> > >
>>>> > >
>>>> > >
>>>> > > And then setup the KSP with
>>>> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>>> > >
>>>> > > The matrix QTBNQ does not change, only the rhs of the system is
>>>> updated.
>>>> > >
>>>> > > We run this with `-pc_type gamg -pc_gamg_type agg
>>>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>>>> > >
>>>> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>>> > >
>>>> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>>> > >
>>>> > > and the same code diverges after 1 timestep and returns a -8
>>>> KSP_DIVERGED_INDEFINITE_PC
>>>> > >
>>>> > > This is weird, especially because if we change nsmooths to 2, it
>>>> runs for 264 timesteps and the returns the same error.  But we have
>>>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>>>> right?
>>>> > >
>>>> > > Change nsmooths to 3 and it again diverges after 1 timestep.
>>>> > >
>>>> > > Change nsmooths to 4 and it runs to completion.
>>>> > >
>>>> > > It seems like either gamg's behavior has changed, or that
>>>> KSPSetNullSpace was doing something implicitly that we now need to do
>>>> explicitly in addition to MatSetNullSpace?
>>>> > >
>>>> > > Thanks,
>>>> > > Gil Forsyth
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> > > -- Norbert Wiener
>>>> > >
>>>> > > <kspview.log>
>>>> >
>>>> >
>>>> > <kspview3.5.4><kspview3.6.1>
>>>>
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150930/fa7dec4d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpi_n2_indefinite_pc.log
Type: text/x-log
Size: 9834 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150930/fa7dec4d/attachment-0001.bin>

From bsmith at mcs.anl.gov  Wed Sep 30 15:24:34 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 30 Sep 2015 15:24:34 -0500
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
Message-ID: <F8632F16-0A1F-41FA-9EE3-B1748BF4BA56@mcs.anl.gov>


  Did the exact same thing run in parallel without the indefinite problem? 

  Run that failure with -info and send all the output

   You could use bisection to find out exactly what change in the library breaks your example.

  Barry

> On Sep 30, 2015, at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> 
> Using PETSc master branch solved the problem in serial, but I'm still seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This runs to completion when I don't use GAMG.  Log is attached for the following run.
> 
> $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2 $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
> 
> 
> Thanks again,
> Gil Forsyth
> 
> 
> On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> Ah, got it.  I'll checkout the master branch and see if the behavior persists.  
> 
> Many thanks,
> Gil
> 
> On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that the behavior seems to have changed between versions.  The only difference in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to MatSetNullSpace.
> 
> Mark made some GAMG changes which were later reversed because they had unintended consequences like this.
> I think what Barry means is, "you should get the behavior you expect using the master branch from PETSc development"
> 
>   Thanks,
> 
>      Matt
>  
> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Update your PETSc
> 
> 
> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> >
> > Hi Barry,
> >
> > We aren't explicitly setting GMRES anywhere in the code and I'm not sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace works with:
> >
> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason > kspview3.5.4
> >
> > and shows that the coarse grid solver is of type:preonly
> >
> > running the newer version that uses MatSetNullSpace in its stead and adding in -poisson_mg_coarse_ksp_type preonly
> >
> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason > kspview3.6.1
> >
> > still shows
> >
> > KSP Object:(poisson_) 1 MPI processes
> >   type: cg
> >   maximum iterations=10000
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >   left preconditioning
> >   using nonzero initial guess
> >   using PRECONDITIONED norm type for convergence test
> > PC Object:(poisson_) 1 MPI processes
> >   type: gamg
> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
> >       Cycles per PCApply=1
> >       Using Galerkin computed coarse grid matrices
> >       GAMG specific options
> >         Threshold for dropping small values from graph 0
> >         AGG specific options
> >           Symmetric graph false
> >   Coarse grid solver -- level -------------------------------
> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
> >       type: gmres
> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> >         GMRES: happy breakdown tolerance 1e-30
> >       maximum iterations=1, initial guess is zero
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >       left preconditioning
> >       using NONE norm type for convergence test
> >
> >
> > both logs are attached.
> >
> >
> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >    This can't work. You can't use a GMRES inside a CG.   Try changing to -poisson_mg_coarse_ksp_type preonly
> >
> > KSP Object:(poisson_) 1 MPI processes
> >   type: cg
> >
> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
> >       type: gmres
> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> >         GMRES: happy breakdown tolerance 1e-30
> >       maximum iterations=1, initial guess is zero
> >
> >
> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > >
> > >
> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > > Hi all,
> > >
> > > I've been having some trouble with what should be a relatively simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
> > >
> > > I'm getting indefinite PC errors for a simple lid-driven cavity test problem, 32x32 at Re 100
> > >
> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set the null space.  This is for a 2D Poisson system with no immersed boundary and so the null space is the constant vector.
> > >
> > >   MatNullSpace nsp;
> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL, &nsp); CHKERRQ(ierr);
> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
> > >
> > > Clearly this has to happen in the reverse order, since ksp2 would not be created yet.
> > >
> > > For questions about solvers, we HAVE to see the complete output of -ksp_view so we
> > > know what we are dealing with. Its also nice to have -ksp_monitor_true_residual -ksp_converged_reason
> > >
> > >   Matt
> > >
> > > Yes -- sorry, those are both in inline files and are called in the reverse order that I wrote them out.
> > >
> > > I've attached the output of
> > >
> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason > kspview.log
> > >
> > >
> > >
> > > And then setup the KSP with
> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
> > >
> > > The matrix QTBNQ does not change, only the rhs of the system is updated.
> > >
> > > We run this with `-pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
> > >
> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
> > >
> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
> > >
> > > and the same code diverges after 1 timestep and returns a -8 KSP_DIVERGED_INDEFINITE_PC
> > >
> > > This is weird, especially because if we change nsmooths to 2, it runs for 264 timesteps and the returns the same error.  But we have explicitly set KSPSetReusePreconditioner so it should be using the same PC, right?
> > >
> > > Change nsmooths to 3 and it again diverges after 1 timestep.
> > >
> > > Change nsmooths to 4 and it runs to completion.
> > >
> > > It seems like either gamg's behavior has changed, or that KSPSetNullSpace was doing something implicitly that we now need to do explicitly in addition to MatSetNullSpace?
> > >
> > > Thanks,
> > > Gil Forsyth
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > > -- Norbert Wiener
> > >
> > > <kspview.log>
> >
> >
> > <kspview3.5.4><kspview3.6.1>
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> <mpi_n2_indefinite_pc.log>


From gforsyth at gwu.edu  Wed Sep 30 15:36:03 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Wed, 30 Sep 2015 16:36:03 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <F8632F16-0A1F-41FA-9EE3-B1748BF4BA56@mcs.anl.gov>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
	<F8632F16-0A1F-41FA-9EE3-B1748BF4BA56@mcs.anl.gov>
Message-ID: <CAHDu6k5h_wweUpUVbZW=V_c1X=XypUYwFD=YCU8o9XgbM5b1AA@mail.gmail.com>

The exact same thing ran in serial without the indefinite problem, but it
does crop up in all parallel runs.

I've attached the failure run log and I'll start bisecting against 3.5.4 to
try to track down the change.

Thanks!
Gil Forsyth

On Wed, Sep 30, 2015 at 4:24 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Did the exact same thing run in parallel without the indefinite problem?
>
>   Run that failure with -info and send all the output
>
>    You could use bisection to find out exactly what change in the library
> breaks your example.
>
>   Barry
>
> > On Sep 30, 2015, at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> >
> > Using PETSc master branch solved the problem in serial, but I'm still
> seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
> runs to completion when I don't use GAMG.  Log is attached for the
> following run.
> >
> > $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2
> $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
> >
> >
> > Thanks again,
> > Gil Forsyth
> >
> >
> > On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > Ah, got it.  I'll checkout the master branch and see if the behavior
> persists.
> >
> > Many thanks,
> > Gil
> >
> > On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that
> the behavior seems to have changed between versions.  The only difference
> in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
> MatSetNullSpace.
> >
> > Mark made some GAMG changes which were later reversed because they had
> unintended consequences like this.
> > I think what Barry means is, "you should get the behavior you expect
> using the master branch from PETSc development"
> >
> >   Thanks,
> >
> >      Matt
> >
> > On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Update your PETSc
> >
> >
> > > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > >
> > > Hi Barry,
> > >
> > > We aren't explicitly setting GMRES anywhere in the code and I'm not
> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
> works with:
> > >
> > > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
> kspview3.5.4
> > >
> > > and shows that the coarse grid solver is of type:preonly
> > >
> > > running the newer version that uses MatSetNullSpace in its stead and
> adding in -poisson_mg_coarse_ksp_type preonly
> > >
> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
> kspview3.6.1
> > >
> > > still shows
> > >
> > > KSP Object:(poisson_) 1 MPI processes
> > >   type: cg
> > >   maximum iterations=10000
> > >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> > >   left preconditioning
> > >   using nonzero initial guess
> > >   using PRECONDITIONED norm type for convergence test
> > > PC Object:(poisson_) 1 MPI processes
> > >   type: gamg
> > >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
> > >       Cycles per PCApply=1
> > >       Using Galerkin computed coarse grid matrices
> > >       GAMG specific options
> > >         Threshold for dropping small values from graph 0
> > >         AGG specific options
> > >           Symmetric graph false
> > >   Coarse grid solver -- level -------------------------------
> > >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
> > >       type: gmres
> > >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > >         GMRES: happy breakdown tolerance 1e-30
> > >       maximum iterations=1, initial guess is zero
> > >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> > >       left preconditioning
> > >       using NONE norm type for convergence test
> > >
> > >
> > > both logs are attached.
> > >
> > >
> > > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >    This can't work. You can't use a GMRES inside a CG.   Try changing
> to -poisson_mg_coarse_ksp_type preonly
> > >
> > > KSP Object:(poisson_) 1 MPI processes
> > >   type: cg
> > >
> > > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
> > >       type: gmres
> > >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > >         GMRES: happy breakdown tolerance 1e-30
> > >       maximum iterations=1, initial guess is zero
> > >
> > >
> > > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > > >
> > > >
> > > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
> wrote:
> > > > Hi all,
> > > >
> > > > I've been having some trouble with what should be a relatively
> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
> > > >
> > > > I'm getting indefinite PC errors for a simple lid-driven cavity test
> problem, 32x32 at Re 100
> > > >
> > > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set
> the null space.  This is for a 2D Poisson system with no immersed boundary
> and so the null space is the constant vector.
> > > >
> > > >   MatNullSpace nsp;
> > > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
> &nsp); CHKERRQ(ierr);
> > > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
> > > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
> > > >
> > > > Clearly this has to happen in the reverse order, since ksp2 would
> not be created yet.
> > > >
> > > > For questions about solvers, we HAVE to see the complete output of
> -ksp_view so we
> > > > know what we are dealing with. Its also nice to have
> -ksp_monitor_true_residual -ksp_converged_reason
> > > >
> > > >   Matt
> > > >
> > > > Yes -- sorry, those are both in inline files and are called in the
> reverse order that I wrote them out.
> > > >
> > > > I've attached the output of
> > > >
> > > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
> -poisson_ksp_view -poisson_ksp_monitor_true_residual
> -poisson_ksp_converged_reason > kspview.log
> > > >
> > > >
> > > >
> > > > And then setup the KSP with
> > > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
> > > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
> > > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
> > > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> > > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
> > > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> > > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
> > > >
> > > > The matrix QTBNQ does not change, only the rhs of the system is
> updated.
> > > >
> > > > We run this with `-pc_type gamg -pc_gamg_type agg
> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
> > > >
> > > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
> > > >
> > > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
> > > >
> > > > and the same code diverges after 1 timestep and returns a -8
> KSP_DIVERGED_INDEFINITE_PC
> > > >
> > > > This is weird, especially because if we change nsmooths to 2, it
> runs for 264 timesteps and the returns the same error.  But we have
> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
> right?
> > > >
> > > > Change nsmooths to 3 and it again diverges after 1 timestep.
> > > >
> > > > Change nsmooths to 4 and it runs to completion.
> > > >
> > > > It seems like either gamg's behavior has changed, or that
> KSPSetNullSpace was doing something implicitly that we now need to do
> explicitly in addition to MatSetNullSpace?
> > > >
> > > > Thanks,
> > > > Gil Forsyth
> > > >
> > > >
> > > >
> > > > --
> > > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > > -- Norbert Wiener
> > > >
> > > > <kspview.log>
> > >
> > >
> > > <kspview3.5.4><kspview3.6.1>
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> >
> > <mpi_n2_indefinite_pc.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150930/10226405/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fullinfo.log
Type: text/x-log
Size: 172844 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150930/10226405/attachment-0001.bin>

From matt.landreman at gmail.com  Wed Sep 30 16:10:38 2015
From: matt.landreman at gmail.com (Matt Landreman)
Date: Wed, 30 Sep 2015 17:10:38 -0400
Subject: [petsc-users] Automatically re-solving after MUMPS error
In-Reply-To: <E589862D-F077-4233-A0DD-48F9257F32C3@mcs.anl.gov>
References: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
	<CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>
	<CAJCWK9BihEkqFefFMPjQNBj5gzp6Zfh-r_gdh=Mh+SL5GJcD5A@mail.gmail.com>
	<CADZ9mvkNH3CvwZigy5cccX5q8oPeHOvDVxPn52q3J3wvn00xWQ@mail.gmail.com>
	<E589862D-F077-4233-A0DD-48F9257F32C3@mcs.anl.gov>
Message-ID: <CADZ9mvkJM02+BsSkAtrsrfHXPD2WEKzkeqDR_1_WR40Mm3mfDA@mail.gmail.com>

Hi Barry,
I tried adding PetscMallocDump after SNESDestroy as you suggested. When
mumps fails, PetscMallocDump shows a number of mallocs which are absent
when mumps succeeds, the largest being MatConvertToTriples_mpiaij_mpiaij()
(line 638 in petsc-3.6.0/src/mat/impls/aij/mpi/mumps/mumps.c).  The total
memory reported by PetscMallocDump after SNESDestroy is substantially
(>20x) larger when mumps fails than when mumps succeeds, and this amount
increases uniformly with each mumps failure.  So I think some of the
mumps-related structures are not being deallocated by SNESDestroy if mumps
generates an error.
Thanks,
-Matt

On Wed, Sep 30, 2015 at 2:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Sep 30, 2015, at 1:06 PM, Matt Landreman <matt.landreman at gmail.com>
> wrote:
> >
> > PETSc developers,
> >
> > I tried implementing a system for automatically increasing MUMPS
> ICNTL(14), along the lines described in this recent thread. If SNESSolve
> returns ierr .ne. 0 due to MUMPS error -9, I call SNESDestroy,
> re-initialize SNES, call MatMumpsSetIcntl with a larger value of ICNTL(14),
> call SNESSolve again, and repeat as needed. The procedure works, but the
> peak memory required (as measured by the HPC system) is 50%-100% higher if
> the MUMPS solve has to be repeated compared to when MUMPS works on the 1st
> try (by starting with a large ICNTL(14)), even though SNESDestroy is called
> in between the attempts. Are there some PETSc or MUMPS structures which
> would not be deallocated immediately by SNESDestroy?  If so, how do I
> deallocate them?
>
>    They should be all destroyed automatically for you. You can use
> PetscMallocDump() after the SNES is destroyed to verify that all that
> memory is not properly freed.
>
>    My guess is that your new malloc() with the bigger workspace cannot
> "reuse" the space that was previously freed; so to the OS it looks like you
> are using a lot more space but in terms of physical memory you are not
> using more.
>
>   Barry
>
> >
> > Thanks,
> > Matt Landreman
> >
> >
> > On Tue, Sep 15, 2015 at 7:47 AM, David Knezevic <
> david.knezevic at akselos.com> wrote:
> > On Tue, Sep 15, 2015 at 7:29 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Tue, Sep 15, 2015 at 4:30 AM, David Knezevic <
> david.knezevic at akselos.com> wrote:
> > In some cases, I get MUMPS error -9, i.e.:
> > [2]PETSC ERROR: Error reported by MUMPS in numerical factorization
> phase: INFO(1)=-9, INFO(2)=98927
> >
> > This is easily fixed by re-running the executable with
> -mat_mumps_icntl_14 on the commandline.
> >
> > However, I would like to update my code in order to do this
> automatically, i.e. detect the -9 error and re-run with the appropriate
> option. Is there a recommended way to do this? It seems to me that I could
> do this with a PETSc error handler (e.g. PetscPushErrorHandler) in order to
> call a function that sets the appropriate option and solves again, is that
> right? Are there any examples that illustrate this type of thing?
> >
> > I would not use the error handler. I would just check the ierr return
> code from the solver. I think you need the
> > INFO output, for which you can use MatMumpsGetInfo().
> >
> >
> > OK, that sounds good (and much simpler than what I had in mind), thanks
> for the help!
> >
> > David
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150930/953bb945/attachment.html>

From bsmith at mcs.anl.gov  Wed Sep 30 16:59:20 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 30 Sep 2015 16:59:20 -0500
Subject: [petsc-users] Automatically re-solving after MUMPS error
In-Reply-To: <CADZ9mvkJM02+BsSkAtrsrfHXPD2WEKzkeqDR_1_WR40Mm3mfDA@mail.gmail.com>
References: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
	<CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>
	<CAJCWK9BihEkqFefFMPjQNBj5gzp6Zfh-r_gdh=Mh+SL5GJcD5A@mail.gmail.com>
	<CADZ9mvkNH3CvwZigy5cccX5q8oPeHOvDVxPn52q3J3wvn00xWQ@mail.gmail.com>
	<E589862D-F077-4233-A0DD-48F9257F32C3@mcs.anl.gov>
	<CADZ9mvkJM02+BsSkAtrsrfHXPD2WEKzkeqDR_1_WR40Mm3mfDA@mail.gmail.com>
Message-ID: <0E0EDF20-41C6-4C5D-A32E-837F0FCE38D6@mcs.anl.gov>


  Matt,

   Yes, you must be right The MatDestroy() on the "partially factored" matrix should clean up everything properly but it sounds like it is not. I'll look at it right now but I only have a few minutes; if I can't resolve it really quickly it may take a day or two.


  Barry

> On Sep 30, 2015, at 4:10 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
> 
> Hi Barry,
> I tried adding PetscMallocDump after SNESDestroy as you suggested. When mumps fails, PetscMallocDump shows a number of mallocs which are absent when mumps succeeds, the largest being MatConvertToTriples_mpiaij_mpiaij() (line 638 in petsc-3.6.0/src/mat/impls/aij/mpi/mumps/mumps.c).  The total memory reported by PetscMallocDump after SNESDestroy is substantially (>20x) larger when mumps fails than when mumps succeeds, and this amount increases uniformly with each mumps failure.  So I think some of the mumps-related structures are not being deallocated by SNESDestroy if mumps generates an error.
> Thanks,
> -Matt
> 
> On Wed, Sep 30, 2015 at 2:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Sep 30, 2015, at 1:06 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
> >
> > PETSc developers,
> >
> > I tried implementing a system for automatically increasing MUMPS ICNTL(14), along the lines described in this recent thread. If SNESSolve returns ierr .ne. 0 due to MUMPS error -9, I call SNESDestroy, re-initialize SNES, call MatMumpsSetIcntl with a larger value of ICNTL(14), call SNESSolve again, and repeat as needed. The procedure works, but the peak memory required (as measured by the HPC system) is 50%-100% higher if the MUMPS solve has to be repeated compared to when MUMPS works on the 1st try (by starting with a large ICNTL(14)), even though SNESDestroy is called in between the attempts. Are there some PETSc or MUMPS structures which would not be deallocated immediately by SNESDestroy?  If so, how do I deallocate them?
> 
>    They should be all destroyed automatically for you. You can use PetscMallocDump() after the SNES is destroyed to verify that all that memory is not properly freed.
> 
>    My guess is that your new malloc() with the bigger workspace cannot "reuse" the space that was previously freed; so to the OS it looks like you are using a lot more space but in terms of physical memory you are not using more.
> 
>   Barry
> 
> >
> > Thanks,
> > Matt Landreman
> >
> >
> > On Tue, Sep 15, 2015 at 7:47 AM, David Knezevic <david.knezevic at akselos.com> wrote:
> > On Tue, Sep 15, 2015 at 7:29 PM, Matthew Knepley <knepley at gmail.com> wrote:
> > On Tue, Sep 15, 2015 at 4:30 AM, David Knezevic <david.knezevic at akselos.com> wrote:
> > In some cases, I get MUMPS error -9, i.e.:
> > [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-9, INFO(2)=98927
> >
> > This is easily fixed by re-running the executable with -mat_mumps_icntl_14 on the commandline.
> >
> > However, I would like to update my code in order to do this automatically, i.e. detect the -9 error and re-run with the appropriate option. Is there a recommended way to do this? It seems to me that I could do this with a PETSc error handler (e.g. PetscPushErrorHandler) in order to call a function that sets the appropriate option and solves again, is that right? Are there any examples that illustrate this type of thing?
> >
> > I would not use the error handler. I would just check the ierr return code from the solver. I think you need the
> > INFO output, for which you can use MatMumpsGetInfo().
> >
> >
> > OK, that sounds good (and much simpler than what I had in mind), thanks for the help!
> >
> > David
> >
> >
> 
> 


From bsmith at mcs.anl.gov  Wed Sep 30 17:28:07 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 30 Sep 2015 17:28:07 -0500
Subject: [petsc-users] Automatically re-solving after MUMPS error
In-Reply-To: <0E0EDF20-41C6-4C5D-A32E-837F0FCE38D6@mcs.anl.gov>
References: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
	<CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>
	<CAJCWK9BihEkqFefFMPjQNBj5gzp6Zfh-r_gdh=Mh+SL5GJcD5A@mail.gmail.com>
	<CADZ9mvkNH3CvwZigy5cccX5q8oPeHOvDVxPn52q3J3wvn00xWQ@mail.gmail.com>
	<E589862D-F077-4233-A0DD-48F9257F32C3@mcs.anl.gov>
	<CADZ9mvkJM02+BsSkAtrsrfHXPD2WEKzkeqDR_1_WR40Mm3mfDA@mail.gmail.com>
	<0E0EDF20-41C6-4C5D-A32E-837F0FCE38D6@mcs.anl.gov>
Message-ID: <FA18A1A8-D8D3-4B84-BBDA-6DDE76E84ABC@mcs.anl.gov>


  Matt,

   Please try the following: edit 

#undef __FUNCT__
#define __FUNCT__ "MatDestroy_MUMPS"
PetscErrorCode MatDestroy_MUMPS(Mat A)
{
  Mat_MUMPS      *mumps=(Mat_MUMPS*)A->spptr;
  PetscErrorCode ierr;

  PetscFunctionBegin;
  if (mumps->CleanUpMUMPS) {

  Remove this if () test and just always do the lines of clean up code after it. Let us know if this resolves the problem?

Thanks

   Barry

This CleanUpMUMPS flag has always be goofy and definitely needs to be removed, the only question is if some other changes are needed when it is removed.


> On Sep 30, 2015, at 4:59 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Matt,
> 
>   Yes, you must be right The MatDestroy() on the "partially factored" matrix should clean up everything properly but it sounds like it is not. I'll look at it right now but I only have a few minutes; if I can't resolve it really quickly it may take a day or two.
> 
> 
>  Barry
> 
>> On Sep 30, 2015, at 4:10 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
>> 
>> Hi Barry,
>> I tried adding PetscMallocDump after SNESDestroy as you suggested. When mumps fails, PetscMallocDump shows a number of mallocs which are absent when mumps succeeds, the largest being MatConvertToTriples_mpiaij_mpiaij() (line 638 in petsc-3.6.0/src/mat/impls/aij/mpi/mumps/mumps.c).  The total memory reported by PetscMallocDump after SNESDestroy is substantially (>20x) larger when mumps fails than when mumps succeeds, and this amount increases uniformly with each mumps failure.  So I think some of the mumps-related structures are not being deallocated by SNESDestroy if mumps generates an error.
>> Thanks,
>> -Matt
>> 
>> On Wed, Sep 30, 2015 at 2:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>>> On Sep 30, 2015, at 1:06 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
>>> 
>>> PETSc developers,
>>> 
>>> I tried implementing a system for automatically increasing MUMPS ICNTL(14), along the lines described in this recent thread. If SNESSolve returns ierr .ne. 0 due to MUMPS error -9, I call SNESDestroy, re-initialize SNES, call MatMumpsSetIcntl with a larger value of ICNTL(14), call SNESSolve again, and repeat as needed. The procedure works, but the peak memory required (as measured by the HPC system) is 50%-100% higher if the MUMPS solve has to be repeated compared to when MUMPS works on the 1st try (by starting with a large ICNTL(14)), even though SNESDestroy is called in between the attempts. Are there some PETSc or MUMPS structures which would not be deallocated immediately by SNESDestroy?  If so, how do I deallocate them?
>> 
>>   They should be all destroyed automatically for you. You can use PetscMallocDump() after the SNES is destroyed to verify that all that memory is not properly freed.
>> 
>>   My guess is that your new malloc() with the bigger workspace cannot "reuse" the space that was previously freed; so to the OS it looks like you are using a lot more space but in terms of physical memory you are not using more.
>> 
>>  Barry
>> 
>>> 
>>> Thanks,
>>> Matt Landreman
>>> 
>>> 
>>> On Tue, Sep 15, 2015 at 7:47 AM, David Knezevic <david.knezevic at akselos.com> wrote:
>>> On Tue, Sep 15, 2015 at 7:29 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>> On Tue, Sep 15, 2015 at 4:30 AM, David Knezevic <david.knezevic at akselos.com> wrote:
>>> In some cases, I get MUMPS error -9, i.e.:
>>> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-9, INFO(2)=98927
>>> 
>>> This is easily fixed by re-running the executable with -mat_mumps_icntl_14 on the commandline.
>>> 
>>> However, I would like to update my code in order to do this automatically, i.e. detect the -9 error and re-run with the appropriate option. Is there a recommended way to do this? It seems to me that I could do this with a PETSc error handler (e.g. PetscPushErrorHandler) in order to call a function that sets the appropriate option and solves again, is that right? Are there any examples that illustrate this type of thing?
>>> 
>>> I would not use the error handler. I would just check the ierr return code from the solver. I think you need the
>>> INFO output, for which you can use MatMumpsGetInfo().
>>> 
>>> 
>>> OK, that sounds good (and much simpler than what I had in mind), thanks for the help!
>>> 
>>> David
>>> 
>>> 
>> 
>> 
>