[petsc-users] pcfieldsplit for a composite dm with multiple subfields

[Matthew Knepley]

On Tue, Sep 8, 2015 at 9:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   yeah but the crappy nonlinear convergence could easily be do to a bad
> Jacobian and 9 times out of 10 bad Newton convergence comes from a bad
> Jacobian.
>

Oh, I thought this was the -snes_fd run. I am getting confused. I think we
need to see -snes_view every time and
it should tell us what Jaociban was used.

   Matt


> > On Sep 8, 2015, at 9:53 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Tue, Sep 8, 2015 at 9:48 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> > Your comment about the different pieces scaling different made me think
> of something, and I made a slight modification.  It still fails, but the
> conditioning looks much better, no?
> >
> > This is great. Now its clear that the nonlinear convergence is crappy.
> >
> > Lets start with looking at the distribution of residuals. Take iterate
> 50,
> > which is as converged as you get and save the residual (VecView with
> > binary). Then we can load it up and play with it. I recommend
> >
> >   a) Norms for both halves
> >
> >   b) Plot of the big half
> >
> > What usually happens is that some small part is screwing up, but you
> > have to iterate the whole system, and finding the descent direction is
> > hard. If you select the problem variables, Newton will converge, and then
> > the big system will converge once the problem is removed. This is the
> classic
> > "coarse basis selection" problem in the nonlinear regime.
> >
> >    Matt
> >
> >   Thanks,
> >
> >     Matt
> >
> >
> >       0 SNES Function norm 4.027474333769e-09
> >         0 KSP preconditioned resid norm 9.390682304142e-09 true resid
> norm 4.027474333769e-09 ||r(i)||/||b|| 1.000000000000e+00
> >         1 KSP preconditioned resid norm 4.937459249116e-20 true resid
> norm 3.737772204700e-19 ||r(i)||/||b|| 9.280685350023e-11
> >       1 SNES Function norm 2.666655509899e-12
> >     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations
> 1
> >     0 SNES Function norm 5.066222213175e+03
> >       0 KSP preconditioned resid norm 3.135087050015e+01 true resid norm
> 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.220576053756e-11 true resid norm
> 1.206309587478e-07 ||r(i)||/||b|| 2.381083056997e-11
> >     1 SNES Function norm 8.484697184819e+02
> >       0 KSP preconditioned resid norm 2.668083368240e+02 true resid norm
> 8.484697184819e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 8.388166746665e-10 true resid norm
> 2.735732864000e-08 ||r(i)||/||b|| 3.224314085004e-11
> >     2 SNES Function norm 6.549559723821e+02
> >       0 KSP preconditioned resid norm 1.552154273063e+02 true resid norm
> 6.549559723821e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.740227244709e-10 true resid norm
> 1.800722435354e-08 ||r(i)||/||b|| 2.749379364852e-11
> >     3 SNES Function norm 5.770723278635e+02
> >       0 KSP preconditioned resid norm 1.348596653013e+02 true resid norm
> 5.770723278635e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.142365551700e-10 true resid norm
> 7.387664544933e-09 ||r(i)||/||b|| 1.280197331985e-11
> >     4 SNES Function norm 5.237702241066e+02
> >       0 KSP preconditioned resid norm 1.524401223470e+02 true resid norm
> 5.237702241066e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.168592748079e-10 true resid norm
> 9.512653260875e-09 ||r(i)||/||b|| 1.816188248788e-11
> >     5 SNES Function norm 4.753909020302e+02
> >       0 KSP preconditioned resid norm 1.864261890160e+02 true resid norm
> 4.753909020302e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.744912816503e-10 true resid norm
> 1.889384889937e-08 ||r(i)||/||b|| 3.974381675939e-11
> >     6 SNES Function norm 4.221784591168e+02
> >       0 KSP preconditioned resid norm 2.550992140633e+02 true resid norm
> 4.221784591168e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.697718851690e-10 true resid norm
> 1.666255394677e-08 ||r(i)||/||b|| 3.946803439861e-11
> >     7 SNES Function norm 3.806525080909e+02
> >       0 KSP preconditioned resid norm 5.183605288372e+02 true resid norm
> 3.806525080909e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.240268561213e-09 true resid norm
> 1.041310587019e-07 ||r(i)||/||b|| 2.735593657958e-10
> >     8 SNES Function norm 3.762054656363e+02
> >       0 KSP preconditioned resid norm 5.320776249886e+03 true resid norm
> 3.762054656363e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.328856456273e-09 true resid norm
> 7.860313076537e-08 ||r(i)||/||b|| 2.089367059897e-10
> >     9 SNES Function norm 3.758975227207e+02
> >       0 KSP preconditioned resid norm 6.252711400154e+03 true resid norm
> 3.758975227207e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.900963654627e-09 true resid norm
> 1.347299589292e-07 ||r(i)||/||b|| 3.584220453330e-10
> >    10 SNES Function norm 3.757032042008e+02
> >       0 KSP preconditioned resid norm 5.890497230519e+02 true resid norm
> 3.757032042008e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.497822807230e-09 true resid norm
> 3.605311757144e-08 ||r(i)||/||b|| 9.596169840535e-11
> >    11 SNES Function norm 3.728798164183e+02
> >       0 KSP preconditioned resid norm 1.081356468099e+03 true resid norm
> 3.728798164183e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.713971332514e-09 true resid norm
> 3.705072372867e-08 ||r(i)||/||b|| 9.936371478768e-11
> >    12 SNES Function norm 3.723078736486e+02
> >       0 KSP preconditioned resid norm 3.745123875883e+03 true resid norm
> 3.723078736486e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.885886422519e-08 true resid norm
> 2.837206039485e-07 ||r(i)||/||b|| 7.620591022374e-10
> >    13 SNES Function norm 3.721848051981e+02
> >       0 KSP preconditioned resid norm 3.495480420016e+03 true resid norm
> 3.721848051981e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.874759607493e-08 true resid norm
> 2.751735241072e-07 ||r(i)||/||b|| 7.393464759014e-10
> >    14 SNES Function norm 3.720227571463e+02
> >       0 KSP preconditioned resid norm 1.265998097646e+04 true resid norm
> 3.720227571463e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.125038941589e-08 true resid norm
> 5.141611324046e-07 ||r(i)||/||b|| 1.382069033488e-09
> >    15 SNES Function norm 3.720051993666e+02
> >       0 KSP preconditioned resid norm 2.912369751490e+03 true resid norm
> 3.720051993666e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.170907138699e-09 true resid norm
> 3.623872061163e-08 ||r(i)||/||b|| 9.741455407971e-11
> >    16 SNES Function norm 3.718945615434e+02
> >       0 KSP preconditioned resid norm 1.082053028195e+03 true resid norm
> 3.718945615434e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.660982704773e-09 true resid norm
> 9.903133228018e-08 ||r(i)||/||b|| 2.662887348210e-10
> >    17 SNES Function norm 3.700412140239e+02
> >       0 KSP preconditioned resid norm 3.676306553630e+01 true resid norm
> 3.700412140239e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.389666775474e-10 true resid norm
> 4.384613797840e-10 ||r(i)||/||b|| 1.184898771183e-12
> >    18 SNES Function norm 3.351961239413e+02
> >       0 KSP preconditioned resid norm 1.071519127966e+02 true resid norm
> 3.351961239413e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.391049839274e-11 true resid norm
> 1.771280721778e-09 ||r(i)||/||b|| 5.284311468017e-12
> >    19 SNES Function norm 3.096040290055e+02
> >       0 KSP preconditioned resid norm 3.362840348158e+02 true resid norm
> 3.096040290055e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.847034649159e-10 true resid norm
> 3.697317355249e-09 ||r(i)||/||b|| 1.194208411023e-11
> >    20 SNES Function norm 3.008687462984e+02
> >       0 KSP preconditioned resid norm 7.161404453567e+01 true resid norm
> 3.008687462984e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.352839847013e-11 true resid norm
> 7.837384020140e-10 ||r(i)||/||b|| 2.604917963917e-12
> >    21 SNES Function norm 2.752608806757e+02
> >       0 KSP preconditioned resid norm 1.727563200247e+02 true resid norm
> 2.752608806757e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.663026698374e-10 true resid norm
> 5.807801532466e-09 ||r(i)||/||b|| 2.109926233692e-11
> >    22 SNES Function norm 2.707559177016e+02
> >       0 KSP preconditioned resid norm 4.125046548626e+02 true resid norm
> 2.707559177016e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.109007703208e-10 true resid norm
> 5.875389803064e-09 ||r(i)||/||b|| 2.169994972940e-11
> >    23 SNES Function norm 2.698855074973e+02
> >       0 KSP preconditioned resid norm 7.414116220947e+01 true resid norm
> 2.698855074973e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.222534270039e-11 true resid norm
> 6.928234778488e-10 ||r(i)||/||b|| 2.567101450810e-12
> >    24 SNES Function norm 2.616892369202e+02
> >       0 KSP preconditioned resid norm 1.819161466540e+02 true resid norm
> 2.616892369202e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.013662651542e-09 true resid norm
> 4.195842561304e-09 ||r(i)||/||b|| 1.603368411588e-11
> >    25 SNES Function norm 2.600248325147e+02
> >       0 KSP preconditioned resid norm 1.265123744997e+02 true resid norm
> 2.600248325147e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.572744639399e-09 true resid norm
> 7.386966838391e-09 ||r(i)||/||b|| 2.840869761150e-11
> >    26 SNES Function norm 2.593065715720e+02
> >       0 KSP preconditioned resid norm 1.314965938655e+02 true resid norm
> 2.593065715720e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.156889834113e-10 true resid norm
> 3.157778817609e-09 ||r(i)||/||b|| 1.217778168314e-11
> >    27 SNES Function norm 2.517348459439e+02
> >       0 KSP preconditioned resid norm 2.618005944700e+01 true resid norm
> 2.517348459439e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 8.483924363849e-11 true resid norm
> 7.418968110702e-10 ||r(i)||/||b|| 2.947135936976e-12
> >    28 SNES Function norm 2.298635352641e+02
> >       0 KSP preconditioned resid norm 8.770042104744e+01 true resid norm
> 2.298635352641e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.098440763657e-09 true resid norm
> 3.936819814733e-09 ||r(i)||/||b|| 1.712676962968e-11
> >    29 SNES Function norm 2.269871118080e+02
> >       0 KSP preconditioned resid norm 2.285655807501e+02 true resid norm
> 2.269871118080e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.216116049541e-08 true resid norm
> 9.327420966977e-08 ||r(i)||/||b|| 4.109229327024e-10
> >    30 SNES Function norm 2.252105956612e+02
> >       0 KSP preconditioned resid norm 8.597156834231e+01 true resid norm
> 2.252105956612e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.431615405953e-09 true resid norm
> 4.297181487549e-09 ||r(i)||/||b|| 1.908072519826e-11
> >    31 SNES Function norm 2.251883212710e+02
> >       0 KSP preconditioned resid norm 7.066983917118e+01 true resid norm
> 2.251883212710e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.998573955346e-10 true resid norm
> 3.454505271056e-09 ||r(i)||/||b|| 1.534051700176e-11
> >    32 SNES Function norm 2.117895362999e+02
> >       0 KSP preconditioned resid norm 3.830985324064e+02 true resid norm
> 2.117895362999e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.016573825110e-09 true resid norm
> 1.942657883557e-08 ||r(i)||/||b|| 9.172586698552e-11
> >    33 SNES Function norm 2.116404150058e+02
> >       0 KSP preconditioned resid norm 5.329232125733e+02 true resid norm
> 2.116404150058e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.609537995048e-09 true resid norm
> 1.451505196923e-08 ||r(i)||/||b|| 6.858355465253e-11
> >    34 SNES Function norm 2.115566314991e+02
> >       0 KSP preconditioned resid norm 6.609724236205e+02 true resid norm
> 2.115566314991e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.254105497370e-10 true resid norm
> 1.178528765891e-08 ||r(i)||/||b|| 5.570748397436e-11
> >    35 SNES Function norm 2.115028944726e+02
> >       0 KSP preconditioned resid norm 8.217040556045e+02 true resid norm
> 2.115028944726e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.803324342385e-09 true resid norm
> 2.957366914143e-08 ||r(i)||/||b|| 1.398263093050e-10
> >    36 SNES Function norm 2.114773510911e+02
> >       0 KSP preconditioned resid norm 1.091957913212e+03 true resid norm
> 2.114773510911e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.264170340401e-08 true resid norm
> 1.635833495717e-07 ||r(i)||/||b|| 7.735265678696e-10
> >    37 SNES Function norm 2.114573155596e+02
> >       0 KSP preconditioned resid norm 1.295967660499e+03 true resid norm
> 2.114573155596e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.787403648651e-08 true resid norm
> 1.913077973453e-07 ||r(i)||/||b|| 9.047111793648e-10
> >    38 SNES Function norm 2.114434864633e+02
> >       0 KSP preconditioned resid norm 1.567323979910e+03 true resid norm
> 2.114434864633e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.869044911030e-09 true resid norm
> 3.425540409553e-08 ||r(i)||/||b|| 1.620073744928e-10
> >    39 SNES Function norm 2.114337906450e+02
> >       0 KSP preconditioned resid norm 1.835099219771e+03 true resid norm
> 2.114337906450e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.224665285731e-10 true resid norm
> 1.690151657152e-08 ||r(i)||/||b|| 7.993763210677e-11
> >    40 SNES Function norm 2.114267348757e+02
> >       0 KSP preconditioned resid norm 2.146249211809e+03 true resid norm
> 2.114267348757e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.809299780773e-08 true resid norm
> 1.157600089098e-07 ||r(i)||/||b|| 5.475183116166e-10
> >    41 SNES Function norm 2.114215910347e+02
> >       0 KSP preconditioned resid norm 2.510692990698e+03 true resid norm
> 2.114215910347e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.485943641401e-08 true resid norm
> 9.439839396991e-08 ||r(i)||/||b|| 4.464936315535e-10
> >    42 SNES Function norm 2.114206676373e+02
> >       0 KSP preconditioned resid norm 3.401544811763e+03 true resid norm
> 2.114206676373e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.910138749741e-09 true resid norm
> 3.806926432018e-08 ||r(i)||/||b|| 1.800640625423e-10
> >    43 SNES Function norm 2.114186309697e+02
> >       0 KSP preconditioned resid norm 3.955962524189e+03 true resid norm
> 2.114186309697e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.500201400572e-08 true resid norm
> 1.570197988244e-07 ||r(i)||/||b|| 7.426961290224e-10
> >    44 SNES Function norm 2.114171338084e+02
> >       0 KSP preconditioned resid norm 4.636713819556e+03 true resid norm
> 2.114171338084e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.248004040828e-09 true resid norm
> 7.296845022145e-08 ||r(i)||/||b|| 3.451397193171e-10
> >    45 SNES Function norm 2.114170015592e+02
> >       0 KSP preconditioned resid norm 6.321621979244e+03 true resid norm
> 2.114170015592e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.674310076597e-08 true resid norm
> 1.788726595879e-07 ||r(i)||/||b|| 8.460656345928e-10
> >    46 SNES Function norm 2.114164190412e+02
> >       0 KSP preconditioned resid norm 7.414534654352e+03 true resid norm
> 2.114164190412e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.520165386851e-08 true resid norm
> 4.326674448851e-07 ||r(i)||/||b|| 2.046517705897e-09
> >    47 SNES Function norm 2.114162562951e+02
> >       0 KSP preconditioned resid norm 9.810108530473e+03 true resid norm
> 2.114162562951e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.690981568660e-08 true resid norm
> 1.826808079437e-07 ||r(i)||/||b|| 8.640811787375e-10
> >    48 SNES Function norm 2.114160137794e+02
> >       0 KSP preconditioned resid norm 1.136099973228e+04 true resid norm
> 2.114160137794e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.015775310105e-08 true resid norm
> 1.261980904678e-07 ||r(i)||/||b|| 5.969183138583e-10
> >    49 SNES Function norm 2.114158331092e+02
> >       0 KSP preconditioned resid norm 1.315560026827e+04 true resid norm
> 2.114158331092e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.131165057283e-08 true resid norm
> 3.097361970470e-07 ||r(i)||/||b|| 1.465056767470e-09
> >    50 SNES Function norm 2.114158223286e+02
> >       0 KSP preconditioned resid norm 1.794717411653e+04 true resid norm
> 2.114158223286e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.070352092082e-07 true resid norm
> 6.224398071454e-07 ||r(i)||/||b|| 2.944149592446e-09
> >    51 SNES Function norm 2.114157500757e+02
> >       0 KSP preconditioned resid norm 2.077631649185e+04 true resid norm
> 2.114157500757e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.358541701826e-08 true resid norm
> 5.604873460299e-07 ||r(i)||/||b|| 2.651114431301e-09
> >    52 SNES Function norm 2.114156961871e+02
> >       0 KSP preconditioned resid norm 2.405048247398e+04 true resid norm
> 2.114156961871e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.439555573169e-08 true resid norm
> 3.896791261159e-07 ||r(i)||/||b|| 1.843189191455e-09
> >    53 SNES Function norm 2.114156592651e+02
> >       0 KSP preconditioned resid norm 2.914886710376e+04 true resid norm
> 2.114156592651e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.436687722935e-07 true resid norm
> 1.370484810325e-06 ||r(i)||/||b|| 6.482418639609e-09
> >    54 SNES Function norm 2.114156319126e+02
> >       0 KSP preconditioned resid norm 3.373835363786e+04 true resid norm
> 2.114156319126e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.028533154395e-07 true resid norm
> 6.668879459515e-07 ||r(i)||/||b|| 3.154392794508e-09
> >    55 SNES Function norm 2.114156115019e+02
> >       0 KSP preconditioned resid norm 3.904963797200e+04 true resid norm
> 2.114156115019e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.986152192528e-08 true resid norm
> 4.508900491313e-07 ||r(i)||/||b|| 2.132718799374e-09
> >    56 SNES Function norm 2.114155964342e+02
> >       0 KSP preconditioned resid norm 4.594596937760e+04 true resid norm
> 2.114155964342e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.567731897537e-08 true resid norm
> 5.054454305445e-07 ||r(i)||/||b|| 2.390766996709e-09
> >    57 SNES Function norm 2.114155854345e+02
> >       0 KSP preconditioned resid norm 5.317614782539e+04 true resid norm
> 2.114155854345e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.460625544912e-07 true resid norm
> 9.171801951021e-07 ||r(i)||/||b|| 4.338280894556e-09
> >    58 SNES Function norm 2.114155777135e+02
> >       0 KSP preconditioned resid norm 6.400017583847e+04 true resid norm
> 2.114155777135e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 8.093685259962e-08 true resid norm
> 6.506435082837e-07 ||r(i)||/||b|| 3.077557081273e-09
> >    59 SNES Function norm 2.114155720468e+02
> >       0 KSP preconditioned resid norm 7.406807436961e+04 true resid norm
> 2.114155720468e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.571128588743e-07 true resid norm
> 1.516640238741e-06 ||r(i)||/||b|| 7.173739493538e-09
> >    60 SNES Function norm 2.114155680388e+02
> >       0 KSP preconditioned resid norm 8.892466432276e+04 true resid norm
> 2.114155680388e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.335222631846e-08 true resid norm
> 8.161361513986e-07 ||r(i)||/||b|| 3.860340839464e-09
> >    61 SNES Function norm 2.114155651044e+02
> >       0 KSP preconditioned resid norm 1.029106627628e+05 true resid norm
> 2.114155651044e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.244108037270e-07 true resid norm
> 3.009616926997e-06 ||r(i)||/||b|| 1.423555037450e-08
> >    62 SNES Function norm 2.114155629136e+02
> >       0 KSP preconditioned resid norm 1.190953464375e+05 true resid norm
> 2.114155629136e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 5.108162336627e-08 true resid norm
> 9.193387836453e-07 ||r(i)||/||b|| 4.348491525295e-09
> >    63 SNES Function norm 2.114155612779e+02
> >       0 KSP preconditioned resid norm 1.378243866677e+05 true resid norm
> 2.114155612779e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.138603793608e-07 true resid norm
> 2.482322226118e-06 ||r(i)||/||b|| 1.174143573497e-08
> >    64 SNES Function norm 2.114155602404e+02
> >       0 KSP preconditioned resid norm 1.698559088929e+05 true resid norm
> 2.114155602404e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.608494949489e-08 true resid norm
> 1.125534374356e-06 ||r(i)||/||b|| 5.323801015765e-09
> >    65 SNES Function norm 2.114155594364e+02
> >       0 KSP preconditioned resid norm 1.965648734846e+05 true resid norm
> 2.114155594364e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.689853079655e-07 true resid norm
> 2.539748843840e-06 ||r(i)||/||b|| 1.201306493529e-08
> >    66 SNES Function norm 2.114155588361e+02
> >       0 KSP preconditioned resid norm 2.274726625349e+05 true resid norm
> 2.114155588361e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 8.937782777025e-07 true resid norm
> 5.126870538363e-06 ||r(i)||/||b|| 2.425020450996e-08
> >    67 SNES Function norm 2.114155583879e+02
> >       0 KSP preconditioned resid norm 2.632394448333e+05 true resid norm
> 2.114155583879e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.229084794159e-07 true resid norm
> 3.026195587359e-06 ||r(i)||/||b|| 1.431396823599e-08
> >    68 SNES Function norm 2.114155581765e+02
> >       0 KSP preconditioned resid norm 3.367643572924e+05 true resid norm
> 2.114155581765e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.632794900620e-08 true resid norm
> 2.446819364813e-06 ||r(i)||/||b|| 1.157350663271e-08
> >    69 SNES Function norm 2.114155579720e+02
> >       0 KSP preconditioned resid norm 3.897127303190e+05 true resid norm
> 2.114155579720e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.933486330809e-06 true resid norm
> 2.730183810468e-05 ||r(i)||/||b|| 1.291382638372e-07
> >    70 SNES Function norm 2.114155578193e+02
> >       0 KSP preconditioned resid norm 4.509848215072e+05 true resid norm
> 2.114155578193e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.706389710731e-08 true resid norm
> 3.785217910694e-06 ||r(i)||/||b|| 1.790415970205e-08
> >    71 SNES Function norm 2.114155577054e+02
> >       0 KSP preconditioned resid norm 5.242625554614e+05 true resid norm
> 2.114155577054e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.321749665453e-06 true resid norm
> 9.069981074375e-06 ||r(i)||/||b|| 4.290119976419e-08
> >    72 SNES Function norm 2.114155576210e+02
> >       0 KSP preconditioned resid norm 6.066871925199e+05 true resid norm
> 2.114155576210e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 3.327204488687e-07 true resid norm
> 7.090997425424e-06 ||r(i)||/||b|| 3.354056581842e-08
> >    73 SNES Function norm 2.114155575580e+02
> >       0 KSP preconditioned resid norm 7.020695777561e+05 true resid norm
> 2.114155575580e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.123816847857e-06 true resid norm
> 8.125899637337e-06 ||r(i)||/||b|| 3.843567489166e-08
> >    74 SNES Function norm 2.114155575109e+02
> >       0 KSP preconditioned resid norm 8.124474305522e+05 true resid norm
> 2.114155575109e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.293550435673e-07 true resid norm
> 7.844829319667e-06 ||r(i)||/||b|| 3.710620643072e-08
> >    75 SNES Function norm 2.114155574865e+02
> >       0 KSP preconditioned resid norm 1.029314742970e+06 true resid norm
> 2.114155574865e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.033392384072e-07 true resid norm
> 9.399925240813e-06 ||r(i)||/||b|| 4.446184260312e-08
> >    76 SNES Function norm 2.114155574646e+02
> >       0 KSP preconditioned resid norm 1.191137352032e+06 true resid norm
> 2.114155574646e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 4.636526184597e-07 true resid norm
> 1.077765913224e-05 ||r(i)||/||b|| 5.097855267366e-08
> >    77 SNES Function norm 2.114155574575e+02
> >       0 KSP preconditioned resid norm 1.563965951801e+06 true resid norm
> 2.114155574575e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.137216998536e-06 true resid norm
> 5.108848518118e-05 ||r(i)||/||b|| 2.416496013612e-07
> >    78 SNES Function norm 2.114155574558e+02
> >       0 KSP preconditioned resid norm 2.108895892585e+06 true resid norm
> 2.114155574558e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.138935971807e-05 true resid norm
> 6.467637231732e-05 ||r(i)||/||b|| 3.059205911601e-07
> >    79 SNES Function norm 2.114155574511e+02
> >       0 KSP preconditioned resid norm 2.564542138115e+06 true resid norm
> 2.114155574511e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.391129540980e-06 true resid norm
> 4.283623913226e-05 ||r(i)||/||b|| 2.026163052933e-07
> >    80 SNES Function norm 2.114155574475e+02
> >       0 KSP preconditioned resid norm 2.967732587697e+06 true resid norm
> 2.114155574475e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 7.446508619620e-06 true resid norm
> 5.164837794317e-05 ||r(i)||/||b|| 2.442979058246e-07
> >    81 SNES Function norm 2.114155574449e+02
> >       0 KSP preconditioned resid norm 3.434290246612e+06 true resid norm
> 2.114155574449e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.781979196562e-06 true resid norm
> 4.886329177979e-05 ||r(i)||/||b|| 2.311243901364e-07
> >    82 SNES Function norm 2.114155574429e+02
> >       0 KSP preconditioned resid norm 3.974205574181e+06 true resid norm
> 2.114155574429e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.425678262963e-05 true resid norm
> 8.517034218173e-05 ||r(i)||/||b|| 4.028574964485e-07
> >    83 SNES Function norm 2.114155574416e+02
> >       0 KSP preconditioned resid norm 4.785116089506e+06 true resid norm
> 2.114155574416e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.130667094145e-05 true resid norm
> 1.264608296603e-04 ||r(i)||/||b|| 5.981623641639e-07
> >    84 SNES Function norm 2.114155574406e+02
> >       0 KSP preconditioned resid norm 5.537350742612e+06 true resid norm
> 2.114155574406e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.299149833543e-06 true resid norm
> 5.419308321331e-05 ||r(i)||/||b|| 2.563344148812e-07
> >    85 SNES Function norm 2.114155574399e+02
> >       0 KSP preconditioned resid norm 6.767161611151e+06 true resid norm
> 2.114155574399e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.781986133631e-05 true resid norm
> 1.232616969009e-04 ||r(i)||/||b|| 5.830303994346e-07
> >    86 SNES Function norm 2.114155574394e+02
> >       0 KSP preconditioned resid norm 7.867884017146e+06 true resid norm
> 2.114155574394e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.002696664504e-05 true resid norm
> 1.398533205346e-04 ||r(i)||/||b|| 6.615091255747e-07
> >    87 SNES Function norm 2.114155574390e+02
> >       0 KSP preconditioned resid norm 9.102289060164e+06 true resid norm
> 2.114155574390e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.231201322143e-05 true resid norm
> 1.368407286352e-04 ||r(i)||/||b|| 6.472595030036e-07
> >    88 SNES Function norm 2.114155574387e+02
> >       0 KSP preconditioned resid norm 1.063720954841e+07 true resid norm
> 2.114155574387e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.517853677131e-05 true resid norm
> 1.273061147947e-04 ||r(i)||/||b|| 6.021605805032e-07
> >    89 SNES Function norm 2.114155574385e+02
> >       0 KSP preconditioned resid norm 1.230866787650e+07 true resid norm
> 2.114155574385e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.745229646688e-05 true resid norm
> 1.389443348166e-04 ||r(i)||/||b|| 6.572096041560e-07
> >    90 SNES Function norm 2.114155574384e+02
> >       0 KSP preconditioned resid norm 1.436772950622e+07 true resid norm
> 2.114155574384e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 9.955301916175e-05 true resid norm
> 5.633754040143e-04 ||r(i)||/||b|| 2.664777421494e-06
> >    91 SNES Function norm 2.114155574383e+02
> >       0 KSP preconditioned resid norm 1.662010558966e+07 true resid norm
> 2.114155574383e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.147116976734e-05 true resid norm
> 1.351152564949e-04 ||r(i)||/||b|| 6.390979837628e-07
> >    92 SNES Function norm 2.114155574382e+02
> >       0 KSP preconditioned resid norm 1.923255924422e+07 true resid norm
> 2.114155574382e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.030030646485e-05 true resid norm
> 1.877965090103e-04 ||r(i)||/||b|| 8.882814078868e-07
> >    93 SNES Function norm 2.114155574381e+02
> >       0 KSP preconditioned resid norm 2.226793244995e+07 true resid norm
> 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 2.411985876631e-04 true resid norm
> 1.342893408929e-03 ||r(i)||/||b|| 6.351913857247e-06
> >    94 SNES Function norm 2.114155574381e+02
> >       0 KSP preconditioned resid norm 2.576953439486e+07 true resid norm
> 2.114155574381e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 6.147595096922e-05 true resid norm
> 4.578229507611e-04 ||r(i)||/||b|| 2.165512114193e-06
> >    95 SNES Function norm 2.114155574380e+02
> >       0 KSP preconditioned resid norm 2.981662207354e+07 true resid norm
> 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 8.456246888386e-06 true resid norm
> 2.856742906943e-04 ||r(i)||/||b|| 1.351245358460e-06
> >    96 SNES Function norm 2.114155574380e+02
> >       0 KSP preconditioned resid norm 3.451395888978e+07 true resid norm
> 2.114155574380e+02 ||r(i)||/||b|| 1.000000000000e+00
> >       1 KSP preconditioned resid norm 1.972976681165e-05 true resid norm
> 3.126288732572e-04 ||r(i)||/||b|| 1.478741096662e-06
> >   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH
> iterations 96
> >
> > -gideon
> >
> >> On Sep 8, 2015, at 10:31 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >>
> >> On Tue, Sep 8, 2015 at 9:19 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >> Ok, going back and running the problem with the flags, -snes_fd
> -snes_mf_operator -ksp_monitor_true_residual -snes_max_funcs 500000, during
> the grid sequence, I still get:
> >>
> >> Lets try using -pc_type lu instead of MUMPS, just to make sure. If we
> still
> >> get this, then the problem is really horribly conditioned without being
> singular.
> >> We are losing all precision for some reason. Is it possible that the
> two halves
> >> are so differently scaled that we make no progress on one with LU?
> >>
> >>   Thanks,
> >>
> >>      Matt
> >>
> >>       0 SNES Function norm 9.131726454160e-09
> >>         0 KSP preconditioned resid norm 4.201012306235e-09 true resid
> norm 9.131726454160e-09 ||r(i)||/||b|| 1.000000000000e+00
> >>         1 KSP preconditioned resid norm 4.811278613811e-14 true resid
> norm 7.117495013042e-13 ||r(i)||/||b|| 7.794249037978e-05
> >>         2 KSP preconditioned resid norm 1.729322442321e-19 true resid
> norm 2.611605096203e-13 ||r(i)||/||b|| 2.859924800982e-05
> >>       1 SNES Function norm 2.066497600915e-12
> >>     Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE
> iterations 1
> >>     0 SNES Function norm 5.066222213176e+03
> >>       0 KSP preconditioned resid norm 3.135051257976e+01 true resid
> norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.425321207844e-04 true resid
> norm 1.654961474093e+00 ||r(i)||/||b|| 3.266657885218e-04
> >>       2 KSP preconditioned resid norm 2.739130995469e-09 true resid
> norm 7.943827483388e-03 ||r(i)||/||b|| 1.567998234015e-06
> >>     1 SNES Function norm 8.487402654330e+02
> >>       0 KSP preconditioned resid norm 2.670376525819e+02 true resid
> norm 8.487402654330e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 5.518628784729e-02 true resid
> norm 1.820466086409e+00 ||r(i)||/||b|| 2.144903641964e-03
> >>       2 KSP preconditioned resid norm 1.680654269556e-07 true resid
> norm 1.635299027240e-01 ||r(i)||/||b|| 1.926736710678e-04
> >>     2 SNES Function norm 6.547660268557e+02
> >>       0 KSP preconditioned resid norm 1.529984375428e+02 true resid
> norm 6.547660268557e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.106652647444e-02 true resid
> norm 9.350438798847e-01 ||r(i)||/||b|| 1.428058026124e-03
> >>       2 KSP preconditioned resid norm 1.739404692247e-07 true resid
> norm 6.642022841611e-02 ||r(i)||/||b|| 1.014411647701e-04
> >>     3 SNES Function norm 5.771035131885e+02
> >>       0 KSP preconditioned resid norm 1.360697738776e+02 true resid
> norm 5.771035131885e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.323138791370e-03 true resid
> norm 6.419077863137e-02 ||r(i)||/||b|| 1.112292286642e-04
> >>     4 SNES Function norm 5.238679723156e+02
> >>       0 KSP preconditioned resid norm 1.528156345475e+02 true resid
> norm 5.238679723156e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.115694692015e-02 true resid
> norm 1.230231531838e+00 ||r(i)||/||b|| 2.348361795054e-03
> >>       2 KSP preconditioned resid norm 5.444815701049e-08 true resid
> norm 6.640979326953e-02 ||r(i)||/||b|| 1.267681873660e-04
> >>     5 SNES Function norm 4.755828064476e+02
> >>       0 KSP preconditioned resid norm 1.872910183528e+02 true resid
> norm 4.755828064476e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.182180755863e-02 true resid
> norm 5.918459188490e-01 ||r(i)||/||b|| 1.244464498769e-03
> >>       2 KSP preconditioned resid norm 7.426717097996e-08 true resid
> norm 8.473303784961e-02 ||r(i)||/||b|| 1.781667392111e-04
> >>     6 SNES Function norm 4.224257767287e+02
> >>       0 KSP preconditioned resid norm 2.578229460575e+02 true resid
> norm 4.224257767287e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.258348180527e-02 true resid
> norm 3.257118054248e-01 ||r(i)||/||b|| 7.710509712432e-04
> >>       2 KSP preconditioned resid norm 1.122972268815e-07 true resid
> norm 1.274729638407e-01 ||r(i)||/||b|| 3.017641698568e-04
> >>     7 SNES Function norm 3.811223987907e+02
> >>       0 KSP preconditioned resid norm 5.218684581634e+02 true resid
> norm 3.811223987907e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.089005298384e-01 true resid
> norm 5.194667956652e+00 ||r(i)||/||b|| 1.362992039601e-02
> >>       2 KSP preconditioned resid norm 7.570453480881e-07 true resid
> norm 2.863356359719e-01 ||r(i)||/||b|| 7.512957435207e-04
> >>     8 SNES Function norm 3.771290745624e+02
> >>       0 KSP preconditioned resid norm 5.827455360446e+03 true resid
> norm 3.771290745624e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 9.620225652579e-01 true resid
> norm 1.332007773441e+01 ||r(i)||/||b|| 3.531967867995e-02
> >>       2 KSP preconditioned resid norm 3.596801636560e-06 true resid
> norm 3.425465452374e+00 ||r(i)||/||b|| 9.083005483862e-03
> >>     9 SNES Function norm 3.767897776540e+02
> >>       0 KSP preconditioned resid norm 5.185115189077e+03 true resid
> norm 3.767897776540e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 7.021105455600e-01 true resid
> norm 1.043813038739e+01 ||r(i)||/||b|| 2.770279611187e-02
> >>       2 KSP preconditioned resid norm 6.951879108628e-06 true resid
> norm 2.816562934830e+00 ||r(i)||/||b|| 7.475157506572e-03
> >>    10 SNES Function norm 3.746968158051e+02
> >>       0 KSP preconditioned resid norm 8.165048723911e+02 true resid
> norm 3.746968158051e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.177528145230e-01 true resid
> norm 2.394044923481e+00 ||r(i)||/||b|| 6.389285476945e-03
> >>       2 KSP preconditioned resid norm 7.325770200602e-07 true resid
> norm 4.877299135281e-01 ||r(i)||/||b|| 1.301665487816e-03
> >>    11 SNES Function norm 3.726877791108e+02
> >>       0 KSP preconditioned resid norm 5.024547551936e+02 true resid
> norm 3.726877791108e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.640017871568e-01 true resid
> norm 1.648101091752e+00 ||r(i)||/||b|| 4.422203206351e-03
> >>       2 KSP preconditioned resid norm 1.005738633834e-06 true resid
> norm 2.935508517682e-01 ||r(i)||/||b|| 7.876589151074e-04
> >>    12 SNES Function norm 3.668832918639e+02
> >>       0 KSP preconditioned resid norm 7.762104953873e+02 true resid
> norm 3.668832918639e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 9.912259329681e-02 true resid
> norm 1.334600681831e+00 ||r(i)||/||b|| 3.637670919956e-03
> >>       2 KSP preconditioned resid norm 7.640320635302e-07 true resid
> norm 4.471334143387e-01 ||r(i)||/||b|| 1.218734742776e-03
> >>    13 SNES Function norm 3.631997889667e+02
> >>       0 KSP preconditioned resid norm 2.515761606628e+02 true resid
> norm 3.631997889667e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.648297544435e-01 true resid
> norm 2.866637955219e+00 ||r(i)||/||b|| 7.892730233611e-03
> >>       2 KSP preconditioned resid norm 7.305731711999e-07 true resid
> norm 1.472589913546e-01 ||r(i)||/||b|| 4.054490003243e-04
> >>    14 SNES Function norm 3.557844076846e+02
> >>       0 KSP preconditioned resid norm 6.995145168850e+02 true resid
> norm 3.557844076846e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.626527197886e-02 true resid
> norm 5.642316830909e-01 ||r(i)||/||b|| 1.585880861848e-03
> >>       2 KSP preconditioned resid norm 1.040702263139e-06 true resid
> norm 4.073703976052e-01 ||r(i)||/||b|| 1.144992272866e-03
> >>    15 SNES Function norm 3.534147071188e+02
> >>       0 KSP preconditioned resid norm 2.698507093385e+02 true resid
> norm 3.534147071188e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.592052963868e-01 true resid
> norm 1.332346399928e+00 ||r(i)||/||b|| 3.769923472595e-03
> >>       2 KSP preconditioned resid norm 5.021635239169e-07 true resid
> norm 1.573071613558e-01 ||r(i)||/||b|| 4.451064378114e-04
> >>    16 SNES Function norm 3.499900270010e+02
> >>       0 KSP preconditioned resid norm 7.295084253442e+01 true resid
> norm 3.499900270010e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.040248007780e-02 true resid
> norm 1.427506717563e-01 ||r(i)||/||b|| 4.078706841436e-04
> >>       2 KSP preconditioned resid norm 1.246491224109e-07 true resid
> norm 4.074099442269e-02 ||r(i)||/||b|| 1.164061581177e-04
> >>    17 SNES Function norm 3.473017914657e+02
> >>       0 KSP preconditioned resid norm 4.169372990718e+02 true resid
> norm 3.473017914657e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.140552389441e-01 true resid
> norm 1.893314948416e+00 ||r(i)||/||b|| 5.451497789361e-03
> >>       2 KSP preconditioned resid norm 7.751532205539e-07 true resid
> norm 2.494992549211e-01 ||r(i)||/||b|| 7.183932275966e-04
> >>    18 SNES Function norm 3.464416771335e+02
> >>       0 KSP preconditioned resid norm 2.968698304201e+02 true resid
> norm 3.464416771335e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.683113115222e-02 true resid
> norm 2.501212385243e-01 ||r(i)||/||b|| 7.219721385540e-04
> >>       2 KSP preconditioned resid norm 5.274081741342e-07 true resid
> norm 1.712806350336e-01 ||r(i)||/||b|| 4.943996243489e-04
> >>    19 SNES Function norm 3.460747441983e+02
> >>       0 KSP preconditioned resid norm 3.707198448214e+02 true resid
> norm 3.460747441983e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 8.740664131652e-02 true resid
> norm 9.477346570578e-01 ||r(i)||/||b|| 2.738525919461e-03
> >>       2 KSP preconditioned resid norm 4.649501179917e-07 true resid
> norm 2.217443031583e-01 ||r(i)||/||b|| 6.407410736429e-04
> >>    20 SNES Function norm 3.455596398964e+02
> >>       0 KSP preconditioned resid norm 5.000331867417e+02 true resid
> norm 3.455596398964e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 7.821124737573e-02 true resid
> norm 7.651194558377e-01 ||r(i)||/||b|| 2.214145888296e-03
> >>       2 KSP preconditioned resid norm 5.030283955029e-07 true resid
> norm 2.918161448112e-01 ||r(i)||/||b|| 8.444740389784e-04
> >>    21 SNES Function norm 3.453222949611e+02
> >>       0 KSP preconditioned resid norm 7.399233760260e+02 true resid
> norm 3.453222949611e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.030913689264e-01 true resid
> norm 8.930798988128e-01 ||r(i)||/||b|| 2.586221370136e-03
> >>       2 KSP preconditioned resid norm 1.497261432173e-06 true resid
> norm 4.385255355661e-01 ||r(i)||/||b|| 1.269902181136e-03
> >>    22 SNES Function norm 3.452850731057e+02
> >>       0 KSP preconditioned resid norm 1.850020325701e+03 true resid
> norm 3.452850731057e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 8.670597549481e-01 true resid
> norm 6.205758169419e+00 ||r(i)||/||b|| 1.797285389027e-02
> >>       2 KSP preconditioned resid norm 4.122440586032e-06 true resid
> norm 1.109676174925e+00 ||r(i)||/||b|| 3.213797124053e-03
> >>    23 SNES Function norm 3.452722963704e+02
> >>       0 KSP preconditioned resid norm 3.362446711623e+03 true resid
> norm 3.452722963704e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.181412468129e+00 true resid
> norm 1.564006484993e+01 ||r(i)||/||b|| 4.529776936738e-02
> >>       2 KSP preconditioned resid norm 1.165193864469e-05 true resid
> norm 1.869411988347e+00 ||r(i)||/||b|| 5.414312147250e-03
> >>    24 SNES Function norm 3.452679622612e+02
> >>       0 KSP preconditioned resid norm 6.149140486077e+03 true resid
> norm 3.452679622612e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.400761945257e-01 true resid
> norm 3.795886928810e+00 ||r(i)||/||b|| 1.099403171945e-02
> >>       2 KSP preconditioned resid norm 1.754461070240e-06 true resid
> norm 3.648059240080e+00 ||r(i)||/||b|| 1.056587821294e-02
> >>    25 SNES Function norm 3.452667776779e+02
> >>       0 KSP preconditioned resid norm 1.101683177789e+04 true resid
> norm 3.452667776779e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.317993935110e+00 true resid
> norm 2.454711939414e+01 ||r(i)||/||b|| 7.109609432809e-02
> >>       2 KSP preconditioned resid norm 3.457477291813e-05 true resid
> norm 6.807469339003e+00 ||r(i)||/||b|| 1.971654899665e-02
> >>    26 SNES Function norm 3.452666586474e+02
> >>       0 KSP preconditioned resid norm 3.879190850784e+04 true resid
> norm 3.452666586474e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.739562350939e+01 true resid
> norm 1.221587956319e+02 ||r(i)||/||b|| 3.538099974971e-01
> >>       2 KSP preconditioned resid norm 7.455463420492e-05 true resid
> norm 3.080916578781e+01 ||r(i)||/||b|| 8.923295955800e-02
> >>    27 SNES Function norm 3.452666425264e+02
> >>       0 KSP preconditioned resid norm 5.709318799171e+04 true resid
> norm 3.452666425264e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.292899448408e-01 true resid
> norm 2.801354737138e+01 ||r(i)||/||b|| 8.113597990933e-02
> >>    28 SNES Function norm 3.452666374187e+02
> >>       0 KSP preconditioned resid norm 2.488913348294e+05 true resid
> norm 3.452666374187e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 9.691998534170e+02 true resid
> norm 7.244072115265e+03 ||r(i)||/||b|| 2.098109498625e+01
> >>       2 KSP preconditioned resid norm 2.503929513070e-03 true resid
> norm 2.928791518636e+03 ||r(i)||/||b|| 8.482694825461e+00
> >>    29 SNES Function norm 3.452666374177e+02
> >>       0 KSP preconditioned resid norm 2.533150641403e+05 true resid
> norm 3.452666374177e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.817060595281e+02 true resid
> norm 1.333479168158e+03 ||r(i)||/||b|| 3.862172082803e+00
> >>       2 KSP preconditioned resid norm 8.930133492283e-04 true resid
> norm 5.412764197416e+02 ||r(i)||/||b|| 1.567705538507e+00
> >>    30 SNES Function norm 3.452666373128e+02
> >>       0 KSP preconditioned resid norm 2.169409532487e+05 true resid
> norm 3.452666373128e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.325438416139e+02 true resid
> norm 9.441618139691e+02 ||r(i)||/||b|| 2.734587451940e+00
> >>       2 KSP preconditioned resid norm 8.796591247860e-04 true resid
> norm 2.978002325996e+02 ||r(i)||/||b|| 8.625224693509e-01
> >>    31 SNES Function norm 3.452666372449e+02
> >>       0 KSP preconditioned resid norm 1.459475274582e+05 true resid
> norm 3.452666372449e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.978987062939e+02 true resid
> norm 1.411576222768e+03 ||r(i)||/||b|| 4.088365542735e+00
> >>       2 KSP preconditioned resid norm 1.235854958341e-03 true resid
> norm 3.579390387115e+02 ||r(i)||/||b|| 1.036703231936e+00
> >>    32 SNES Function norm 3.452666369928e+02
> >>       0 KSP preconditioned resid norm 1.264234515603e+05 true resid
> norm 3.452666369928e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.676975931928e+01 true resid
> norm 1.855746501202e+02 ||r(i)||/||b|| 5.374821376792e-01
> >>       2 KSP preconditioned resid norm 9.286140031498e-05 true resid
> norm 3.495715551553e+01 ||r(i)||/||b|| 1.012468387331e-01
> >>    33 SNES Function norm 3.452666245901e+02
> >>       0 KSP preconditioned resid norm 1.544591438108e+06 true resid
> norm 3.452666245901e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.501087626485e+01 true resid
> norm 3.233291920369e+02 ||r(i)||/||b|| 9.364623424597e-01
> >>       2 KSP preconditioned resid norm 1.867215855867e-04 true resid
> norm 7.114686634655e+01 ||r(i)||/||b|| 2.060635499623e-01
> >>    34 SNES Function norm 3.452666216061e+02
> >>       0 KSP preconditioned resid norm 3.431570281248e+05 true resid
> norm 3.452666216061e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.140176766342e+02 true resid
> norm 1.760366951993e+03 ||r(i)||/||b|| 5.098572644538e+00
> >>       2 KSP preconditioned resid norm 2.324744486242e-03 true resid
> norm 1.080605579953e+03 ||r(i)||/||b|| 3.129771348664e+00
> >>    35 SNES Function norm 3.452666215921e+02
> >>       0 KSP preconditioned resid norm 3.734063885343e+05 true resid
> norm 3.452666215921e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 8.279219133674e+00 true resid
> norm 1.691481367261e+02 ||r(i)||/||b|| 4.899058470990e-01
> >>       2 KSP preconditioned resid norm 4.046466703016e-04 true resid
> norm 1.964786369469e+02 ||r(i)||/||b|| 5.690635139908e-01
> >>    36 SNES Function norm 3.452666212200e+02
> >>       0 KSP preconditioned resid norm 7.952390829283e+05 true resid
> norm 3.452666212200e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 3.707607129733e+01 true resid
> norm 2.806640653978e+02 ||r(i)||/||b|| 8.128908158167e-01
> >>       2 KSP preconditioned resid norm 1.296391885225e-04 true resid
> norm 1.079623998255e+02 ||r(i)||/||b|| 3.126928384910e-01
> >>    37 SNES Function norm 3.452666199636e+02
> >>       0 KSP preconditioned resid norm 9.358626316710e+05 true resid
> norm 3.452666199636e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.722696571742e+01 true resid
> norm 3.251492717238e+02 ||r(i)||/||b|| 9.417338744130e-01
> >>       2 KSP preconditioned resid norm 2.862108374145e-04 true resid
> norm 3.320104402606e+01 ||r(i)||/||b|| 9.616059620694e-02
> >>    38 SNES Function norm 3.452666133867e+02
> >>       0 KSP preconditioned resid norm 8.365871848022e+04 true resid
> norm 3.452666133867e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.482543031390e+02 true resid
> norm 1.772891732407e+03 ||r(i)||/||b|| 5.134848443690e+00
> >>       2 KSP preconditioned resid norm 1.439398367387e-03 true resid
> norm 3.088902452162e+02 ||r(i)||/||b|| 8.946426710257e-01
> >>    39 SNES Function norm 3.452666132332e+02
> >>       0 KSP preconditioned resid norm 8.960112658041e+04 true resid
> norm 3.452666132332e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.505891399150e+01 true resid
> norm 1.787713293837e+02 ||r(i)||/||b|| 5.177776319280e-01
> >>       2 KSP preconditioned resid norm 6.384856870578e-05 true resid
> norm 2.543851208517e+01 ||r(i)||/||b|| 7.367787996341e-02
> >>    40 SNES Function norm 3.452665904125e+02
> >>       0 KSP preconditioned resid norm 7.796808144307e+05 true resid
> norm 3.452665904125e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 9.020764189760e+01 true resid
> norm 6.961545395677e+02 ||r(i)||/||b|| 2.016281212544e+00
> >>       2 KSP preconditioned resid norm 3.506849883654e-04 true resid
> norm 4.956460306859e+02 ||r(i)||/||b|| 1.435545878024e+00
> >>    41 SNES Function norm 3.452665902941e+02
> >>       0 KSP preconditioned resid norm 4.990797566798e+05 true resid
> norm 3.452665902941e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.012324354359e+02 true resid
> norm 7.390530765626e+02 ||r(i)||/||b|| 2.140528789458e+00
> >>       2 KSP preconditioned resid norm 7.111862680292e-04 true resid
> norm 4.066260265930e+02 ||r(i)||/||b|| 1.177716112777e+00
> >>    42 SNES Function norm 3.452665901159e+02
> >>       0 KSP preconditioned resid norm 3.472654025442e+05 true resid
> norm 3.452665901159e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.552680791160e+02 true resid
> norm 1.064105325830e+03 ||r(i)||/||b|| 3.081981739016e+00
> >>       2 KSP preconditioned resid norm 2.695346315896e-03 true resid
> norm 5.359619569435e+02 ||r(i)||/||b|| 1.552313407340e+00
> >>    43 SNES Function norm 3.452665900083e+02
> >>       0 KSP preconditioned resid norm 2.823873507449e+05 true resid
> norm 3.452665900083e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.616335635707e+00 true resid
> norm 1.564312339233e+02 ||r(i)||/||b|| 4.530737651723e-01
> >>       2 KSP preconditioned resid norm 1.580836053583e-04 true resid
> norm 1.777870114826e+02 ||r(i)||/||b|| 5.149267743465e-01
> >>    44 SNES Function norm 3.452665895210e+02
> >>       0 KSP preconditioned resid norm 4.764215602672e+05 true resid
> norm 3.452665895210e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 6.491949190147e+01 true resid
> norm 4.652947485976e+02 ||r(i)||/||b|| 1.347639078670e+00
> >>       2 KSP preconditioned resid norm 2.266106888017e-04 true resid
> norm 1.316722015429e+02 ||r(i)||/||b|| 3.813638664707e-01
> >>    45 SNES Function norm 3.452665889075e+02
> >>       0 KSP preconditioned resid norm 1.439659890301e+05 true resid
> norm 3.452665889075e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.431461947162e+01 true resid
> norm 1.403614551576e+02 ||r(i)||/||b|| 4.065306625865e-01
> >>       2 KSP preconditioned resid norm 4.711117728059e-04 true resid
> norm 1.095117712535e+02 ||r(i)||/||b|| 3.171803318704e-01
> >>    46 SNES Function norm 3.452665876547e+02
> >>       0 KSP preconditioned resid norm 2.157313334766e+05 true resid
> norm 3.452665876547e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 2.967750174401e+00 true resid
> norm 1.053591658730e+02 ||r(i)||/||b|| 3.051530893525e-01
> >>       2 KSP preconditioned resid norm 4.512059841816e-04 true resid
> norm 1.189061020519e+02 ||r(i)||/||b|| 3.443892525473e-01
> >>    47 SNES Function norm 3.452665866229e+02
> >>       0 KSP preconditioned resid norm 3.981118078251e+05 true resid
> norm 3.452665866229e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 7.013119377232e+01 true resid
> norm 4.969789311062e+02 ||r(i)||/||b|| 1.439406390196e+00
> >>       2 KSP preconditioned resid norm 1.872910444347e-04 true resid
> norm 1.396396719776e+02 ||r(i)||/||b|| 4.044401554852e-01
> >>    48 SNES Function norm 3.452665850954e+02
> >>       0 KSP preconditioned resid norm 1.967554630062e+05 true resid
> norm 3.452665850954e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 6.006970327024e+00 true resid
> norm 9.616098906643e+01 ||r(i)||/||b|| 2.785122951874e-01
> >>       2 KSP preconditioned resid norm 3.184368801558e-05 true resid
> norm 9.951979756268e+01 ||r(i)||/||b|| 2.882404549377e-01
> >>    49 SNES Function norm 3.452665835600e+02
> >>       0 KSP preconditioned resid norm 3.965648247310e+05 true resid
> norm 3.452665835600e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.862969730098e+01 true resid
> norm 1.790616593203e+02 ||r(i)||/||b|| 5.186185627174e-01
> >>       2 KSP preconditioned resid norm 2.964457441143e-04 true resid
> norm 1.447818294544e+02 ||r(i)||/||b|| 4.193334552146e-01
> >>    50 SNES Function norm 3.452665828612e+02
> >>       0 KSP preconditioned resid norm 1.265022286683e+06 true resid
> norm 3.452665828612e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 4.401066920177e+01 true resid
> norm 3.143671803790e+02 ||r(i)||/||b|| 9.105056671685e-01
> >>       2 KSP preconditioned resid norm 1.444898193175e-04 true resid
> norm 7.578180059616e+01 ||r(i)||/||b|| 2.194877939480e-01
> >>    51 SNES Function norm 3.452665817118e+02
> >>       0 KSP preconditioned resid norm 2.063334812849e+05 true resid
> norm 3.452665817118e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 1.074628697499e+01 true resid
> norm 1.155569679685e+02 ||r(i)||/||b|| 3.346891187543e-01
> >>       2 KSP preconditioned resid norm 1.326420665890e-04 true resid
> norm 9.261090091148e+01 ||r(i)||/||b|| 2.682301323584e-01
> >>    52 SNES Function norm 3.452665807811e+02
> >>       0 KSP preconditioned resid norm 6.549963238965e+06 true resid
> norm 3.452665807811e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>       1 KSP preconditioned resid norm 5.246352248726e+01 true resid
> norm 3.604213542889e+02 ||r(i)||/||b|| 1.043892963731e+00
> >>   Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH
> iterations 52
> >>
> >> which is a little different, but not so much different, than what i get
> if I use my Jacobian.
> >>
> >> -gideon
> >>
> >>> On Sep 8, 2015, at 10:00 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> >>>
> >>> Yup, now it’s working.
> >>> -gideon
> >>>
> >>>> On Sep 8, 2015, at 9:58 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> >>>>
> >>>> On Tue, Sep 8, 2015 at 8:51 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>> Going back to this point, a case that previously worked, is now
> giving me problems that I don’t understand:
> >>>>
> >>>>   0 SNES Function norm 1.421454390131e-02
> >>>>     0 KSP preconditioned resid norm 7.635339584553e-03 true resid
> norm 1.421454390131e-02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>     1 KSP preconditioned resid norm 2.838956718317e-07 true resid
> norm 1.299176431150e-06 ||r(i)||/||b|| 9.139768677554e-05
> >>>>     2 KSP preconditioned resid norm 5.889748199467e-12 true resid
> norm 9.554723457719e-07 ||r(i)||/||b|| 6.721793906335e-05
> >>>>   1 SNES Function norm 4.476139939604e-06
> >>>>     0 KSP preconditioned resid norm 4.897102387844e-05 true resid
> norm 4.476139939604e-06 ||r(i)||/||b|| 1.000000000000e+00
> >>>>     1 KSP preconditioned resid norm 1.133649519960e-09 true resid
> norm 5.352604068352e-09 ||r(i)||/||b|| 1.195808026687e-03
> >>>>     2 KSP preconditioned resid norm 2.223328042462e-14 true resid
> norm 6.113509642967e-09 ||r(i)||/||b|| 1.365799489171e-03
> >>>>   2 SNES Function norm 5.447761863330e-09
> >>>>     0 KSP preconditioned resid norm 9.703890346186e-09 true resid
> norm 5.447761863330e-09 ||r(i)||/||b|| 1.000000000000e+00
> >>>>     1 KSP preconditioned resid norm 1.982949185218e-13 true resid
> norm 1.118082736179e-12 ||r(i)||/||b|| 2.052370797823e-04
> >>>>     2 KSP preconditioned resid norm 7.045835741315e-18 true resid
> norm 1.249234275881e-12 ||r(i)||/||b|| 2.293114690439e-04
> >>>> Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT
> iterations 2
> >>>>
> >>>> Somehow the maximum number of function evaluations is being exceeded.
> Oh, are you using -snes_fd? This uses a function evaluation
> >>>> for each column, so you might have to up it > 10000. The convergence
> is fine.
> >>>>
> >>>>   Thanks,
> >>>>
> >>>>      Matt
> >>>>
> >>>> -gideon
> >>>>
> >>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >>>>>
> >>>>>
> >>>>>   You can run with -snes_fd it will be slow so run with only two
> levels of grid sequencing; check the convergence rate of Newton on the
> finer grid. It will hopefully be much better (also KSP should converge in
> one iteration). Send the output
> >>>>>
> >>>>>  Barry
> >>>>>
> >>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>
> >>>>>> I went through my code and nothing jumps out at me as clearly
> wrong.  Is there a way to dump to disk vectors and matrices that would be
> useful for the comparison?  I imagine I’d want the jacobian that I am
> specifying, but what would I want to compare that against in terms of using
> -snes_mf_operator?
> >>>>>>
> >>>>>>
> >>>>>> -gideon
> >>>>>>
> >>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>> Just to clarify, again, during the linear solve phase, even though
> the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that
> the true residual norm could be comparatively large signified there is
> something wrong.  The one thing i was curious about is why it seems that
> there is only an issue during grid sequencing.  I’m not seeing that kind of
> behavior on the coarsest mesh.
> >>>>>>>
> >>>>>>> That definitely sounds like a bug in the Jacobian evaluation.
> Maybe you assume a certain mesh size, and when
> >>>>>>> PETSc refines the DA and passes in the new one something goes
> wrong?
> >>>>>>>
> >>>>>>>   Matt
> >>>>>>>
> >>>>>>> -gideon
> >>>>>>>
> >>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Alright, I’ll take another look at what has been implemented in
> my Jacobian file.  One thing I’m a bit unclear about is do I need these
> -snes_mf and snes_mf_operator flags if I have, coded in,
> >>>>>>>>>
> >>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
> >>>>>>>>
> >>>>>>>> Gideon,
> >>>>>>>>
> >>>>>>>>  In your case the DM creates that Jacobian matrix object and
> form_function_jacobian's job is just to fill it up with the correct values
> for the current solution value.   If you use -snes_mf_operator then SNES
> uses a matrix-free multiple for the first matrix and your Jacobian to build
> the preconditioner. With -snes_mf it uses a matrix-free multiple for the
> first and second matrix and ignores what you compute. Since it is matrix
> free in both it cannot build a preconditioner hence the -pc_type none
> >>>>>>>>
> >>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the
> Jacobian you provide is correct
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Barry
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -gideon
> >>>>>>>>>
> >>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with
> a PC type of LU should give two iterations (or at most three) to get very
> good (10^-12) convergence of the linear system. Here you are not getting
> that after a few SNES iterations. So I am strongly leaning to something
> wrong with the Jacobian you provide.  -snes_type test unfortunately doesn't
> catch all problems.
> >>>>>>>>>>
> >>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear
> solver converges.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Barry
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Off the top of my head, I can’t rule out a null space in this
> problem. I thought I did a pretty good job of coding the Jacobian by hand.
> And before I made that change to the Jacobian file, changing J for Jpre, if
> I ran with  -snes_check_jacobian, it appeared to be satisfactory.
> >>>>>>>>>>>
> >>>>>>>>>>> When I run with -ksp_type fgmres, I get
> >>>>>>>>>>>
> >>>>>>>>>>>   0 SNES Function norm 3.857327250267e-09
> >>>>>>>>>>>     0 KSP unpreconditioned resid norm 3.857327250267e-09 true
> resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>     1 KSP unpreconditioned resid norm 4.490987252203e-13 true
> resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
> >>>>>>>>>>>     2 KSP unpreconditioned resid norm 7.379711327483e-17 true
> resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
> >>>>>>>>>>>   1 SNES Function norm 2.479697868412e-12
> >>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE
> iterations 1
> >>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.066222213176e+03 true
> resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 8.556950606195e-03 true
> resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
> >>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 8.484696862318e+02 true
> resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.384145175257e-01 true
> resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.495757031958e-04 true
> resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
> >>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 6.551139809777e+02 true
> resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.699223578306e-02 true
> resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.336876642454e-05 true
> resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
> >>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.771483337231e+02 true
> resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.809574952128e-02 true
> resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.460582114169e-05 true
> resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
> >>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 5.237540245670e+02 true
> resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.634190283952e-02 true
> resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.087123509737e-04 true
> resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
> >>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.753058321803e+02 true
> resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.480046646239e-02 true
> resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.930291355515e-04 true
> resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
> >>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 4.221482124411e+02 true
> resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.021730633747e-01 true
> resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.424059347969e-04 true
> resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
> >>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.803620028964e+02 true
> resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.347601667746e-01 true
> resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.585189594479e-03 true
> resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
> >>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.769519577570e+02 true
> resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.473450825400e+00 true
> resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.512219713818e-02 true
> resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 9.156193230828e-04 true
> resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
> >>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.766389203406e+02 true
> resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.885345065329e+00 true
> resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.541366781733e-03 true
> resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
> >>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.751822489648e+02 true
> resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.504123802743e-01 true
> resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 3.266605004942e-03 true
> resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
> >>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737189963925e+02 true
> resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 7.988643057771e+00 true
> resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 4.764871173758e-03 true
> resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 3.282807486232e-04 true
> resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
> >>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.737110331594e+02 true
> resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 3.129801991140e-01 true
> resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.125627067813e-03 true
> resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
> >>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.684175513154e+02 true
> resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.242155815449e-01 true
> resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.368849704689e-04 true
> resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
> >>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.509930783933e+02 true
> resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.839683743117e-02 true
> resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.044153843783e-06 true
> resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
> >>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 3.162121096565e+02 true
> resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 2.154093218732e-02 true
> resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.294811898520e-06 true
> resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
> >>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.890165052818e+02 true
> resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.746945848272e-02 true
> resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 6.877255989792e-05 true
> resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
> >>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.816929421445e+02 true
> resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.895856197699e-01 true
> resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.103295403077e-02 true
> resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.552844644077e-05 true
> resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
> >>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.814407559738e+02 true
> resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 1.709056815086e-01 true
> resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 7.267004791015e-04 true
> resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
> >>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.810261821835e+02 true
> resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.414127232289e-02 true
> resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 1.134804652646e-04 true
> resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
> >>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755723672077e+02 true
> resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.096136080591e+00 true
> resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 9.111133958404e-01 true
> resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.910677194353e-01 true
> resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
> >>>>>>>>>>>   4 KSP unpreconditioned resid norm 4.788261382489e-02 true
> resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
> >>>>>>>>>>>   5 KSP unpreconditioned resid norm 1.390068885512e-02 true
> resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
> >>>>>>>>>>>   6 KSP unpreconditioned resid norm 2.384477406046e-03 true
> resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
> >>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755720389981e+02 true
> resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 5.357900039856e+00 true
> resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 5.929328578269e-01 true
> resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.086022506600e-01 true
> resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
> >>>>>>>>>>>   4 KSP unpreconditioned resid norm 1.109330548287e-02 true
> resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
> >>>>>>>>>>>   5 KSP unpreconditioned resid norm 3.644882603935e-03 true
> resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
> >>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.950365335797e-04 true
> resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
> >>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755716464881e+02 true
> resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.907844411256e+00 true
> resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.044873964073e-01 true
> resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 4.734069809407e-02 true
> resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
> >>>>>>>>>>>   4 KSP unpreconditioned resid norm 2.325616228250e-03 true
> resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
> >>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755713163168e+02 true
> resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 6.846535958070e+00 true
> resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.481071550890e+00 true
> resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 8.153960511371e-01 true
> resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
> >>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.272907968554e-01 true
> resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
> >>>>>>>>>>>   5 KSP unpreconditioned resid norm 2.568557800847e-01 true
> resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
> >>>>>>>>>>>   6 KSP unpreconditioned resid norm 9.535234538303e-02 true
> resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
> >>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.803098494703e-03 true
> resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
> >>>>>>>>>>>   8 KSP unpreconditioned resid norm 2.151080788386e-04 true
> resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
> >>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02
> >>>>>>>>>>>   0 KSP unpreconditioned resid norm 2.755712363532e+02 true
> resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>   1 KSP unpreconditioned resid norm 4.232622501221e+01 true
> resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
> >>>>>>>>>>>   2 KSP unpreconditioned resid norm 2.491207938618e+01 true
> resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
> >>>>>>>>>>>   3 KSP unpreconditioned resid norm 1.873650490509e+01 true
> resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
> >>>>>>>>>>>   4 KSP unpreconditioned resid norm 5.309932679768e+00 true
> resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
> >>>>>>>>>>>   5 KSP unpreconditioned resid norm 4.024003774177e+00 true
> resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
> >>>>>>>>>>>   6 KSP unpreconditioned resid norm 3.757785755885e+00 true
> resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
> >>>>>>>>>>>   7 KSP unpreconditioned resid norm 3.609856365266e+00 true
> resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
> >>>>>>>>>>>   8 KSP unpreconditioned resid norm 3.599697975830e+00 true
> resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
> >>>>>>>>>>>   9 KSP unpreconditioned resid norm 3.599279687857e+00 true
> resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
> >>>>>>>>>>>  10 KSP unpreconditioned resid norm 3.597803724329e+00 true
> resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
> >>>>>>>>>>>  11 KSP unpreconditioned resid norm 3.596737615999e+00 true
> resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
> >>>>>>>>>>>  12 KSP unpreconditioned resid norm 3.595710841264e+00 true
> resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
> >>>>>>>>>>>  13 KSP unpreconditioned resid norm 3.586196423514e+00 true
> resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
> >>>>>>>>>>>  14 KSP unpreconditioned resid norm 3.435157463728e+00 true
> resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
> >>>>>>>>>>>  15 KSP unpreconditioned resid norm 3.380326213569e+00 true
> resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
> >>>>>>>>>>>  16 KSP unpreconditioned resid norm 3.370184966977e+00 true
> resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
> >>>>>>>>>>>  17 KSP unpreconditioned resid norm 3.362283769411e+00 true
> resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
> >>>>>>>>>>>  18 KSP unpreconditioned resid norm 3.350650432378e+00 true
> resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
> >>>>>>>>>>>  19 KSP unpreconditioned resid norm 3.349048765263e+00 true
> resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
> >>>>>>>>>>>  20 KSP unpreconditioned resid norm 3.348988713038e+00 true
> resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
> >>>>>>>>>>>  21 KSP unpreconditioned resid norm 3.348981727899e+00 true
> resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
> >>>>>>>>>>>  22 KSP unpreconditioned resid norm 3.347278139858e+00 true
> resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
> >>>>>>>>>>>  23 KSP unpreconditioned resid norm 3.219226414932e+00 true
> resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
> >>>>>>>>>>>  24 KSP unpreconditioned resid norm 2.822590852350e+00 true
> resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
> >>>>>>>>>>>  25 KSP unpreconditioned resid norm 2.309519054919e+00 true
> resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
> >>>>>>>>>>>  26 KSP unpreconditioned resid norm 2.227139205444e+00 true
> resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
> >>>>>>>>>>>  27 KSP unpreconditioned resid norm 1.071083203142e+00 true
> resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
> >>>>>>>>>>>  28 KSP unpreconditioned resid norm 9.806030722653e-01 true
> resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
> >>>>>>>>>>>  29 KSP unpreconditioned resid norm 9.664048337082e-01 true
> resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
> >>>>>>>>>>>  30 KSP unpreconditioned resid norm 2.770004820147e+02 true
> resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
> >>>>>>>>>>>  31 KSP unpreconditioned resid norm 1.140110593381e+02 true
> resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
> >>>>>>>>>>>  32 KSP unpreconditioned resid norm 8.236226033426e+01 true
> resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
> >>>>>>>>>>>  33 KSP unpreconditioned resid norm 3.898917963685e+01 true
> resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
> >>>>>>>>>>>  34 KSP unpreconditioned resid norm 1.011205458583e+01 true
> resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
> >>>>>>>>>>>  35 KSP unpreconditioned resid norm 8.616275790199e+00 true
> resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
> >>>>>>>>>>>  36 KSP unpreconditioned resid norm 7.060253104363e+00 true
> resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
> >>>>>>>>>>>  37 KSP unpreconditioned resid norm 6.570242477517e+00 true
> resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
> >>>>>>>>>>>  38 KSP unpreconditioned resid norm 5.572927530093e+00 true
> resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
> >>>>>>>>>>>  39 KSP unpreconditioned resid norm 5.293220961263e+00 true
> resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
> >>>>>>>>>>>  40 KSP unpreconditioned resid norm 5.256809704195e+00 true
> resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
> >>>>>>>>>>>  41 KSP unpreconditioned resid norm 5.190301236286e+00 true
> resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
> >>>>>>>>>>>  42 KSP unpreconditioned resid norm 5.183649726411e+00 true
> resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
> >>>>>>>>>>>  43 KSP unpreconditioned resid norm 5.118281808364e+00 true
> resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
> >>>>>>>>>>>  44 KSP unpreconditioned resid norm 2.885097693426e+00 true
> resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
> >>>>>>>>>>>  45 KSP unpreconditioned resid norm 1.035319192472e+00 true
> resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
> >>>>>>>>>>>  46 KSP unpreconditioned resid norm 5.579863623125e-01 true
> resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
> >>>>>>>>>>>  47 KSP unpreconditioned resid norm 5.120421018240e-01 true
> resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
> >>>>>>>>>>>  48 KSP unpreconditioned resid norm 4.796787069079e-01 true
> resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
> >>>>>>>>>>>  49 KSP unpreconditioned resid norm 4.744592839962e-01 true
> resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
> >>>>>>>>>>>  50 KSP unpreconditioned resid norm 4.744564914486e-01 true
> resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
> >>>>>>>>>>>  51 KSP unpreconditioned resid norm 4.744563271210e-01 true
> resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
> >>>>>>>>>>>  52 KSP unpreconditioned resid norm 4.741552119642e-01 true
> resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
> >>>>>>>>>>>  53 KSP unpreconditioned resid norm 4.260771566579e-01 true
> resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
> >>>>>>>>>>>  54 KSP unpreconditioned resid norm 1.606767026862e-01 true
> resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
> >>>>>>>>>>>  55 KSP unpreconditioned resid norm 1.438586734835e-01 true
> resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
> >>>>>>>>>>>  56 KSP unpreconditioned resid norm 1.406571559968e-01 true
> resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
> >>>>>>>>>>>  57 KSP unpreconditioned resid norm 1.314866726817e-01 true
> resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
> >>>>>>>>>>>  58 KSP unpreconditioned resid norm 1.294094769063e-01 true
> resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
> >>>>>>>>>>>  59 KSP unpreconditioned resid norm 1.157036938799e-01 true
> resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
> >>>>>>>>>>>  60 KSP unpreconditioned resid norm 8.241203625923e+02 true
> resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
> >>>>>>>>>>>  61 KSP unpreconditioned resid norm 5.762204692795e+02 true
> resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
> >>>>>>>>>>>  62 KSP unpreconditioned resid norm 3.985894224287e+02 true
> resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
> >>>>>>>>>>>  63 KSP unpreconditioned resid norm 3.715639719604e+02 true
> resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
> >>>>>>>>>>>  64 KSP unpreconditioned resid norm 3.517510024064e+02 true
> resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
> >>>>>>>>>>>  65 KSP unpreconditioned resid norm 3.505256741334e+02 true
> resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
> >>>>>>>>>>>  66 KSP unpreconditioned resid norm 3.245282609350e+02 true
> resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
> >>>>>>>>>>>  67 KSP unpreconditioned resid norm 3.185546922426e+02 true
> resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
> >>>>>>>>>>>  68 KSP unpreconditioned resid norm 2.935614497897e+02 true
> resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
> >>>>>>>>>>>  69 KSP unpreconditioned resid norm 2.916950130807e+02 true
> resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
> >>>>>>>>>>>  70 KSP unpreconditioned resid norm 5.338203474651e+01 true
> resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
> >>>>>>>>>>>  71 KSP unpreconditioned resid norm 3.842451984147e+01 true
> resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
> >>>>>>>>>>>  72 KSP unpreconditioned resid norm 2.456635578348e+01 true
> resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
> >>>>>>>>>>>  73 KSP unpreconditioned resid norm 1.197940743903e+01 true
> resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
> >>>>>>>>>>>  74 KSP unpreconditioned resid norm 1.094051775599e+01 true
> resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
> >>>>>>>>>>>  75 KSP unpreconditioned resid norm 1.086455030374e+01 true
> resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
> >>>>>>>>>>>  76 KSP unpreconditioned resid norm 1.079928058658e+01 true
> resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
> >>>>>>>>>>>  77 KSP unpreconditioned resid norm 1.075256031213e+01 true
> resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
> >>>>>>>>>>>  78 KSP unpreconditioned resid norm 1.066313589844e+01 true
> resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
> >>>>>>>>>>>  79 KSP unpreconditioned resid norm 1.059506075324e+01 true
> resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
> >>>>>>>>>>>  80 KSP unpreconditioned resid norm 1.047508685484e+01 true
> resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
> >>>>>>>>>>>  81 KSP unpreconditioned resid norm 1.043458037171e+01 true
> resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
> >>>>>>>>>>>  82 KSP unpreconditioned resid norm 1.039892140666e+01 true
> resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
> >>>>>>>>>>>  83 KSP unpreconditioned resid norm 1.037939199767e+01 true
> resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
> >>>>>>>>>>>  84 KSP unpreconditioned resid norm 9.984750042542e+00 true
> resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
> >>>>>>>>>>>  85 KSP unpreconditioned resid norm 9.883677466536e+00 true
> resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
> >>>>>>>>>>>  86 KSP unpreconditioned resid norm 6.619460053034e+00 true
> resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
> >>>>>>>>>>>  87 KSP unpreconditioned resid norm 6.310659447939e+00 true
> resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
> >>>>>>>>>>>  88 KSP unpreconditioned resid norm 5.965222301448e+00 true
> resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
> >>>>>>>>>>>  89 KSP unpreconditioned resid norm 5.937863010270e+00 true
> resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
> >>>>>>>>>>>  90 KSP unpreconditioned resid norm 7.016500581832e+03 true
> resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
> >>>>>>>>>>>  91 KSP unpreconditioned resid norm 5.093503615041e+03 true
> resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
> >>>>>>>>>>>  92 KSP unpreconditioned resid norm 4.619237287853e+03 true
> resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
> >>>>>>>>>>>  93 KSP unpreconditioned resid norm 4.597871280643e+03 true
> resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
> >>>>>>>>>>>  94 KSP unpreconditioned resid norm 3.891645464749e+03 true
> resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
> >>>>>>>>>>>  95 KSP unpreconditioned resid norm 3.200596462200e+03 true
> resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
> >>>>>>>>>>>  96 KSP unpreconditioned resid norm 3.133449271747e+03 true
> resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
> >>>>>>>>>>>  97 KSP unpreconditioned resid norm 3.132571481382e+03 true
> resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
> >>>>>>>>>>>  98 KSP unpreconditioned resid norm 2.765126329282e+03 true
> resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
> >>>>>>>>>>>  99 KSP unpreconditioned resid norm 2.727323406805e+03 true
> resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
> >>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true
> resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
> >>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true
> resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
> >>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true
> resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
> >>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true
> resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
> >>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true
> resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
> >>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true
> resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
> >>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true
> resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
> >>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true
> resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
> >>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true
> resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
> >>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true
> resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
> >>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true
> resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
> >>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true
> resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
> >>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true
> resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
> >>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true
> resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
> >>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true
> resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
> >>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true
> resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
> >>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true
> resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
> >>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true
> resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
> >>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true
> resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
> >>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true
> resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
> >>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true
> resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
> >>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true
> resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
> >>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true
> resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
> >>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true
> resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
> >>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true
> resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
> >>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true
> resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
> >>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true
> resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
> >>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true
> resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
> >>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true
> resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
> >>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true
> resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
> >>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true
> resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
> >>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true
> resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
> >>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true
> resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
> >>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true
> resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
> >>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true
> resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
> >>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true
> resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
> >>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true
> resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
> >>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true
> resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
> >>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true
> resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
> >>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true
> resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
> >>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true
> resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
> >>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true
> resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
> >>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true
> resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
> >>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true
> resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
> >>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true
> resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
> >>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true
> resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
> >>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true
> resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
> >>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true
> resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
> >>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true
> resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
> >>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true
> resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
> >>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true
> resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
> >>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true
> resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
> >>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true
> resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
> >>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true
> resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
> >>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true
> resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
> >>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true
> resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
> >>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true
> resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
> >>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true
> resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
> >>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true
> resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
> >>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true
> resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
> >>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true
> resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
> >>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true
> resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
> >>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true
> resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
> >>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true
> resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
> >>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true
> resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
> >>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true
> resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
> >>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true
> resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
> >>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true
> resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
> >>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true
> resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
> >>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true
> resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
> >>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true
> resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
> >>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true
> resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
> >>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true
> resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
> >>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true
> resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
> >>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true
> resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
> >>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true
> resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
> >>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true
> resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
> >>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true
> resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
> >>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true
> resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
> >>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true
> resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
> >>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true
> resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
> >>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true
> resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
> >>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true
> resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
> >>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true
> resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
> >>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true
> resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
> >>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true
> resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
> >>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true
> resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
> >>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true
> resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
> >>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true
> resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
> >>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true
> resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
> >>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true
> resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
> >>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true
> resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
> >>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true
> resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
> >>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true
> resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
> >>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true
> resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
> >>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true
> resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
> >>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true
> resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
> >>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true
> resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
> >>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true
> resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
> >>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true
> resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
> >>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true
> resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
> >>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true
> resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
> >>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true
> resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
> >>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true
> resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
> >>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true
> resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
> >>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true
> resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
> >>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true
> resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
> >>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true
> resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
> >>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true
> resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
> >>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true
> resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
> >>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true
> resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
> >>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
> iterations 24
> >>>>>>>>>>>
> >>>>>>>>>>> -gideon
> >>>>>>>>>>>
> >>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For
> example with Neumann boundary conditions there is often a nullspace)
> >>>>>>>>>>>>
> >>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same
> output.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Something is screwy with the linear system convergence, could
> be several things including either a null space problem or that your
> Jacobian that you provide is terribly wrong.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Barry
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02
> >>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true
> resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true
> resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
> >>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true
> resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Got it, fixing that, and returning to the original question,
> this is what I now get, when I use those two flags:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>  0 SNES Function norm 1.132185384796e-08
> >>>>>>>>>>>>>    0 KSP preconditioned resid norm 3.846486104194e-09 true
> resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>    1 KSP preconditioned resid norm 2.813743504021e-14 true
> resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
> >>>>>>>>>>>>>  1 SNES Function norm 2.177599365111e-12
> >>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE
> iterations 1
> >>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.135087050041e+01 true
> resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.498184175046e-04 true
> resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
> >>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.667293100105e+02 true
> resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.682642450862e-02 true
> resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.943933962588e-08 true
> resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
> >>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.538341367076e+02 true
> resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.294478035133e-03 true
> resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.879002692918e-08 true
> resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
> >>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.362249760236e+02 true
> resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.565721843210e-02 true
> resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.900043171759e-08 true
> resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
> >>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.533523981591e+02 true
> resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.030969535942e-02 true
> resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.066614364159e-08 true
> resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
> >>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.848133067820e+02 true
> resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.745259702307e-02 true
> resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.045643600156e-07 true
> resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
> >>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.549151165192e+02 true
> resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.794436126782e-02 true
> resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.890094997586e-07 true
> resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
> >>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.225410707531e+02 true
> resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.810631909992e-03 true
> resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
> >>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.256830807263e+03 true
> resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.034356591467e+00 true
> resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.014544916729e-05 true
> resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
> >>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.988264999623e+03 true
> resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.115167379340e-01 true
> resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.515961219888e-06 true
> resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
> >>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.142561025658e+03 true
> resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.746046512685e-02 true
> resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.588755510089e-06 true
> resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
> >>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.435759974780e+03 true
> resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 7.389097536786e-01 true
> resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.785754916084e-06 true
> resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
> >>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.010267719047e+02 true
> resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.404383077173e-01 true
> resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.431973894553e-07 true
> resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
> >>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.697546464095e+02 true
> resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.100811362173e-01 true
> resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.921252511737e-07 true
> resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
> >>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.455211776575e+01 true
> resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.681410748806e-02 true
> resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.060849362790e-07 true
> resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
> >>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.515566740696e+02 true
> resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.959786024091e-02 true
> resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.945498964959e-07 true
> resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
> >>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 7.834551377727e+01 true
> resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.992168699741e-02 true
> resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 3.192091055088e-07 true
> resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
> >>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.384044359119e+02 true
> resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.206851078700e-01 true
> resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.074373330828e-06 true
> resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
> >>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 9.576546391157e+03 true
> resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.670797073017e+00 true
> resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.278475684614e-05 true
> resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
> >>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.381986890301e+04 true
> resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.870471171439e+01 true
> resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.628164409392e-04 true
> resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
> >>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.191539882357e+04 true
> resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.528735548091e+01 true
> resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.890475574699e-05 true
> resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
> >>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.076322725615e+05 true
> resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.990965035897e+01 true
> resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941661213523e-05 true
> resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
> >>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.312500251239e+06 true
> resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.194187039215e+01 true
> resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.998846844269e-04 true
> resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
> >>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022854344216e+05 true
> resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.602644501358e+00 true
> resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.288860381060e-05 true
> resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
> >>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.876191744212e+05 true
> resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.701586788162e+01 true
> resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.533781032398e-04 true
> resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
> >>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.373321357604e+05 true
> resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.804687152206e+02 true
> resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.556287913942e-03 true
> resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
> >>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.292036277396e+05 true
> resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.282180994978e+01 true
> resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.163542587614e-04 true
> resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
> >>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.447152171843e+05 true
> resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.283727214797e+01 true
> resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 4.585761865240e-04 true
> resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
> >>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.785016507221e+05 true
> resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.095374466008e+01 true
> resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 9.617782709610e-05 true
> resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
> >>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.669977278077e+05 true
> resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.767902454807e+01 true
> resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.086718418040e-04 true
> resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
> >>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.136061088127e+05 true
> resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.186362823355e+02 true
> resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.719259676319e-04 true
> resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
> >>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.691271219294e+05 true
> resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.462251245099e+01 true
> resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.183786013104e-04 true
> resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
> >>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.124446369819e+05 true
> resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 8.973907556421e+01 true
> resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 7.374208523915e-04 true
> resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
> >>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.329193031679e+05 true
> resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.183766435760e+01 true
> resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 6.941412108263e-05 true
> resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
> >>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.096972895505e+05 true
> resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.733316558393e+01 true
> resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.068548659549e-04 true
> resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
> >>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.499649833972e+04 true
> resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.096524448730e+01 true
> resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.801709551672e-04 true
> resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
> >>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 8.633609234208e+04 true
> resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.229972441154e+01 true
> resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 5.664496361269e-05 true
> resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
> >>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.344743984989e+05 true
> resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 2.372624802517e+01 true
> resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 2.527665232153e-04 true
> resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
> >>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.756130938265e+05 true
> resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 4.629319937995e+01 true
> resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 8.099878702234e-04 true
> resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
> >>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 2.326428261777e+05 true
> resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 1.756089168490e+01 true
> resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
> >>>>>>>>>>>>>  2 KSP preconditioned resid norm 1.956105557949e-04 true
> resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
> >>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.133259484912e+07 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.296568492777e+01 true
> resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
> >>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 4.124224624453e+07 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.219934148122e+01 true
> resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
> >>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.748707463105e+08 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.291843121908e+01 true
> resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
> >>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 1.022718794881e+09 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314705152175e+01 true
> resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
> >>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.380361366230e+08 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.237699578864e+01 true
> resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
> >>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 6.303672864381e+08 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.789746869236e+01 true
> resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
> >>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 3.428971983722e+09 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.314776104627e+01 true
> resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
> >>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02
> >>>>>>>>>>>>>  0 KSP preconditioned resid norm 5.189185292378e+09 true
> resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
> >>>>>>>>>>>>>  1 KSP preconditioned resid norm 3.191097159135e+01 true
> resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
> >>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH
> iterations 47
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you
> are setting matrix entries with the first Mat argument when you should
> always set them with the second matrix argument. For example if you have
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J.
> Then at the end of the function you should call MatAssemblyBegin/End() on
> jpre then on J if J is not == jpre  see for example
> src/snes/examples/tutorials/ex1.c
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This is a minor glitch we'll get past.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Barry
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> By the way, I tried using a different petsc installation,
> and now, rather than the segmentation fault, I get the following error:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> >>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this
> object type
> >>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
> >>>>>>>>>>>>>>> [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> >>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
> >>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a
> arch-darwin-c-debug named gs_air by gideon Mon Sep  7 21:32:18 2015
> >>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes
> --download-suitesparse=yes --download-superlu=yes
> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes
> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes
> --download-parmetis=yes --download-scalapack=yes
> >>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in
> /opt/petsc-3.5.4/src/mat/interface/matrix.c
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hmm,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since
> it is one process, type
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste
> all output and send it)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator  and any other
> options you normally use
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Barry
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Running with that flag gives me this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to
> ./blowup_batch_refine of pid 16111 on gs_air
> >>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of
> PETSc installed and the option -start_in_debugger noxterm  Once the
> debugger starts up type cont and when it crashes type where or bt  Send all
> output
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Barry
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I’m getting an error with -snes_mf_operator,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV:
> Segmentation Violation, probably memory access out of range
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on
> GNU/linux and Apple Mac OS X to find memory corruption errors
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes,
> recompile, link, and run
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a
> arch-macports named gs_air by gideon Mon Sep  7 21:08:19 2015
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local
> --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries
> --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0
> --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate
> --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local
> --with-superlu-dir=/opt/local --with-metis-dir=/opt/local
> --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local
> --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local
> CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp
> FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp
> F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os
> FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names"
> CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os
> FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports
> --with-mpiexec=mpiexec-mpich-mp
> >>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in
> unknown file
> >>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) -
> process 0
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct
> "enough"), hence PETSc SNES is generating a poor descent direction. You can
> try
> >>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as
> additional arguments. What happens?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Barry
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> No problem Matt, I don’t think we had previously
> discussed that output.  Here is a case where things fail.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09
> >>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12
> >>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to
> CONVERGED_SNORM_RELATIVE iterations 1
> >>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03
> >>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02
> >>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02
> >>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02
> >>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02
> >>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02
> >>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02
> >>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02
> >>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02
> >>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02
> >>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02
> >>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02
> >>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02
> >>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02
> >>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02
> >>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02
> >>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02
> >>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02
> >>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02
> >>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02
> >>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02
> >>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02
> >>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02
> >>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02
> >>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02
> >>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02
> >>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02
> >>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02
> >>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02
> >>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02
> >>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02
> >>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02
> >>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02
> >>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02
> >>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02
> >>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02
> >>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02
> >>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02
> >>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02
> >>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02
> >>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02
> >>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02
> >>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02
> >>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02
> >>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02
> >>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02
> >>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to
> DIVERGED_LINE_SEARCH iterations 45
> >>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03
> >>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03
> >>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03
> >>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03
> >>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03
> >>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03
> >>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03
> >>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03
> >>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03
> >>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03
> >>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03
> >>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03
> >>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03
> >>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03
> >>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03
> >>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03
> >>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03
> >>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03
> >>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03
> >>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03
> >>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03
> >>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03
> >>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03
> >>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03
> >>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03
> >>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03
> >>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03
> >>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03
> >>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to
> DIVERGED_LINE_SEARCH iterations 27
> >>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
> >>>>>>>>>>>>>>>>>>>>> type: newtonls
> >>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function
> evaluations=10000
> >>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50,
> solution=1e-08
> >>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
> >>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
> >>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
> >>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object:   1 MPI processes
> >>>>>>>>>>>>>>>>>>>>> type: bt
> >>>>>>>>>>>>>>>>>>>>> interpolation: cubic
> >>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
> >>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
> >>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08,
> absolute=1.000000e-15, lambda=1.000000e-08
> >>>>>>>>>>>>>>>>>>>>> maximum iterations=40
> >>>>>>>>>>>>>>>>>>>>> KSP Object:   1 MPI processes
> >>>>>>>>>>>>>>>>>>>>> type: gmres
> >>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified)
> Gram-Schmidt Orthogonalization with no iterative refinement
> >>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
> >>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
> >>>>>>>>>>>>>>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50,
> divergence=10000
> >>>>>>>>>>>>>>>>>>>>> left preconditioning
> >>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
> >>>>>>>>>>>>>>>>>>>>> PC Object:   1 MPI processes
> >>>>>>>>>>>>>>>>>>>>> type: lu
> >>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
> >>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
> >>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
> >>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
> >>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
> >>>>>>>>>>>>>>>>>>>>>  Mat Object:           1 MPI processes
> >>>>>>>>>>>>>>>>>>>>>    type: seqaij
> >>>>>>>>>>>>>>>>>>>>>    rows=15991, cols=15991
> >>>>>>>>>>>>>>>>>>>>>    package used to perform factorization: mumps
> >>>>>>>>>>>>>>>>>>>>>    total: nonzeros=255801, allocated nonzeros=255801
> >>>>>>>>>>>>>>>>>>>>>    total number of mallocs used during MatSetValues
> calls =0
> >>>>>>>>>>>>>>>>>>>>>      MUMPS run parameters:
> >>>>>>>>>>>>>>>>>>>>>        SYM (matrix type):                   0
> >>>>>>>>>>>>>>>>>>>>>        PAR (host participation):            1
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(1) (output for error):         6
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(2) (output of diagnostic msg): 0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(3) (output for global info):   0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(4) (level of printing):        0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(5) (input mat struct):         0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(6) (matrix prescaling):        7
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(7) (sequentia matrix ordering):6
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(8) (scalling strategy):        77
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(10) (max num of refinements):  0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(11) (error analysis):          0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(12) (efficiency control):
>            1
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(13) (efficiency control):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(14) (percentage of estimated workspace
> increase): 20
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(18) (input mat struct):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(19) (Shur complement info):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(20) (rhs sparse pattern):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(21) (somumpstion struct):
>               0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(22) (in-core/out-of-core facility):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(23) (max size of memory can be
> allocated locally):0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(24) (detection of null pivot rows):
>            0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(25) (computation of a null space
> basis):          0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(26) (Schur options for rhs or
> solution):          0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(27) (experimental parameter):
>            -8
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(28) (use parallel or sequential
> ordering):        1
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(29) (parallel ordering):
>           0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(30) (user-specified set of entries in
> inv(A)):    0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(31) (factors is discarded in the solve
> phase):    0
> >>>>>>>>>>>>>>>>>>>>>        ICNTL(33) (compute determinant):
>           0
> >>>>>>>>>>>>>>>>>>>>>        CNTL(1) (relative pivoting threshold):
> 0.01
> >>>>>>>>>>>>>>>>>>>>>        CNTL(2) (stopping criterion of refinement):
> 1.49012e-08
> >>>>>>>>>>>>>>>>>>>>>        CNTL(3) (absomumpste pivoting threshold):
>   0
> >>>>>>>>>>>>>>>>>>>>>        CNTL(4) (vamumpse of static pivoting):
>  -1
> >>>>>>>>>>>>>>>>>>>>>        CNTL(5) (fixation for null pivots):         0
> >>>>>>>>>>>>>>>>>>>>>        RINFO(1) (local estimated flops for the
> elimination after analysis):
> >>>>>>>>>>>>>>>>>>>>>          [0] 1.95838e+06
> >>>>>>>>>>>>>>>>>>>>>        RINFO(2) (local estimated flops for the
> assembly after factorization):
> >>>>>>>>>>>>>>>>>>>>>          [0]  143924
> >>>>>>>>>>>>>>>>>>>>>        RINFO(3) (local estimated flops for the
> elimination after factorization):
> >>>>>>>>>>>>>>>>>>>>>          [0]  1.95943e+06
> >>>>>>>>>>>>>>>>>>>>>        INFO(15) (estimated size of (in MB) MUMPS
> internal data for running numerical factorization):
> >>>>>>>>>>>>>>>>>>>>>        [0] 7
> >>>>>>>>>>>>>>>>>>>>>        INFO(16) (size of (in MB) MUMPS internal data
> used during numerical factorization):
> >>>>>>>>>>>>>>>>>>>>>          [0] 7
> >>>>>>>>>>>>>>>>>>>>>        INFO(23) (num of pivots eliminated on this
> processor after factorization):
> >>>>>>>>>>>>>>>>>>>>>          [0] 15991
> >>>>>>>>>>>>>>>>>>>>>        RINFOG(1) (global estimated flops for the
> elimination after analysis): 1.95838e+06
> >>>>>>>>>>>>>>>>>>>>>        RINFOG(2) (global estimated flops for the
> assembly after factorization): 143924
> >>>>>>>>>>>>>>>>>>>>>        RINFOG(3) (global estimated flops for the
> elimination after factorization): 1.95943e+06
> >>>>>>>>>>>>>>>>>>>>>        (RINFOG(12) RINFOG(13))*2^INFOG(34)
> (determinant): (0,0)*(2^0)
> >>>>>>>>>>>>>>>>>>>>>        INFOG(3) (estimated real workspace for
> factors on all processors after analysis): 255801
> >>>>>>>>>>>>>>>>>>>>>        INFOG(4) (estimated integer workspace for
> factors on all processors after analysis): 127874
> >>>>>>>>>>>>>>>>>>>>>        INFOG(5) (estimated maximum front size in the
> complete tree): 11
> >>>>>>>>>>>>>>>>>>>>>        INFOG(6) (number of nodes in the complete
> tree): 3996
> >>>>>>>>>>>>>>>>>>>>>        INFOG(7) (ordering option effectively use
> after analysis): 6
> >>>>>>>>>>>>>>>>>>>>>        INFOG(8) (structural symmetry in percent of
> the permuted matrix after analysis): 86
> >>>>>>>>>>>>>>>>>>>>>        INFOG(9) (total real/complex workspace to
> store the matrix factors after factorization): 255865
> >>>>>>>>>>>>>>>>>>>>>        INFOG(10) (total integer space store the
> matrix factors after factorization): 127890
> >>>>>>>>>>>>>>>>>>>>>        INFOG(11) (order of largest frontal matrix
> after factorization): 11
> >>>>>>>>>>>>>>>>>>>>>        INFOG(12) (number of off-diagonal pivots): 19
> >>>>>>>>>>>>>>>>>>>>>        INFOG(13) (number of delayed pivots after
> factorization): 8
> >>>>>>>>>>>>>>>>>>>>>        INFOG(14) (number of memory compress after
> factorization): 0
> >>>>>>>>>>>>>>>>>>>>>        INFOG(15) (number of steps of iterative
> refinement after solution): 0
> >>>>>>>>>>>>>>>>>>>>>        INFOG(16) (estimated size (in MB) of all
> MUMPS internal data for factorization after analysis: value on the most
> memory consuming processor): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(17) (estimated size of all MUMPS
> internal data for factorization after analysis: sum over all processors): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(18) (size of all MUMPS internal data
> allocated during factorization: value on the most memory consuming
> processor): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(19) (size of all MUMPS internal data
> allocated during factorization: sum over all processors): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(20) (estimated number of entries in the
> factors): 255801
> >>>>>>>>>>>>>>>>>>>>>        INFOG(21) (size in MB of memory effectively
> used during factorization - value on the most memory consuming processor): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(22) (size in MB of memory effectively
> used during factorization - sum over all processors): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(23) (after analysis: value of ICNTL(6)
> effectively used): 0
> >>>>>>>>>>>>>>>>>>>>>        INFOG(24) (after analysis: value of ICNTL(12)
> effectively used): 1
> >>>>>>>>>>>>>>>>>>>>>        INFOG(25) (after factorization: number of
> pivots modified by static pivoting): 0
> >>>>>>>>>>>>>>>>>>>>>        INFOG(28) (after factorization: number of
> null pivots encountered): 0
> >>>>>>>>>>>>>>>>>>>>>        INFOG(29) (after factorization: effective
> number of entries in the factors (sum over all processors)): 255865
> >>>>>>>>>>>>>>>>>>>>>        INFOG(30, 31) (after solution: size in Mbytes
> of memory used during solution phase): 5, 5
> >>>>>>>>>>>>>>>>>>>>>        INFOG(32) (after analysis: type of analysis
> done): 1
> >>>>>>>>>>>>>>>>>>>>>        INFOG(33) (value used for ICNTL(8)): 7
> >>>>>>>>>>>>>>>>>>>>>        INFOG(34) (exponent of the determinant if
> determinant is requested): 0
> >>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
> >>>>>>>>>>>>>>>>>>>>> Mat Object:     1 MPI processes
> >>>>>>>>>>>>>>>>>>>>> type: seqaij
> >>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
> >>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
> >>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues
> calls =15991
> >>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used
> is 5
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <
> knepley at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>> Barry,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid
> sequencing within my code.  I find that, in some cases, even if it can
> solve successfully on the coarsest mesh, the SNES fails, usually due to a
> line search failure, when it tries to compute along the grid sequence.
> Would you have any suggestions?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you
> give me -snes_view for the solver? I could not find it in the email thread.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line
> search, or precondition it with Richardson. It would be nice to see
> -snes_monitor
> >>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down
> the residual into fields and look at it again (if my custom residual monitor
> >>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing
> which part of the residual does not converge is key to designing the NASM
> >>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of
> this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Matt
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <
> gideon.simpson at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the
> coarse mesh, I can generally solve at all values. I do find that I need to
> do some amount of continuation to solve near the endpoint.  The problem is
> that on the coarse mesh, things are not fully resolved at all the values
> along the continuation parameter, and I would like to do refinement.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the
> intermediate continuation solutions  too.  Currently, without doing any
> grid sequence, I compute each, write it to disk, and then go on to the next
> one.  So I now need to go back an refine them.  I was thinking that perhaps
> I could refine them on the fly, dump them to disk, and use the coarse
> solution as the starting guess at the next iteration, but that would seem
> to require resetting the snes back to the coarse grid.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh
> refinement in a post processing stage, where each value of the continuation
> is parameter is loaded on the coarse mesh, and refined.  Perhaps that’s the
> most practical thing to do.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and
> create a SNES that will do the continuation
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined
> solution at this continuation point) {
> >>>>>>>>>>>>>>>>>>>>>>>    SNESCreate(comm,&snesrefine);
> >>>>>>>>>>>>>>>>>>>>>>>    SNESSetDM()
> >>>>>>>>>>>>>>>>>>>>>>>    etc
> >>>>>>>>>>>>>>>>>>>>>>>    SNESSetGridSequence(snesrefine,)
> >>>>>>>>>>>>>>>>>>>>>>>    SNESSolve(snesrefine,0,Ucoarse);
> >>>>>>>>>>>>>>>>>>>>>>>    SNESGetSolution(snesrefine,&Ufine);
> >>>>>>>>>>>>>>>>>>>>>>>    VecView(Ufine or do whatever you want to do
> with the Ufine at that continuation point
> >>>>>>>>>>>>>>>>>>>>>>>    SNESDestroy(snesrefine);
> >>>>>>>>>>>>>>>>>>>>>>> end if
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Barry
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> -gideon
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 3.  This problem is actually part of a
> continuation problem that roughly looks like this
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>      solve with parameter p_i using solution
> from p_{i-1},
> >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver,
> for each value of parameter p_i on the coarse mesh, and then do grid
> sequencing on that.  But it appears that after doing grid sequencing on the
> initial p_0 = 0, the SNES is set to use the finer mesh.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good
> enough initial guess on the coarse level to even get convergence on the
> coarse level? First I would check if you even need the continuation (or can
> you not even solve the coarse problem without it).
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will
> need to tweak how you do the grid sequencing. I think this will work:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with
> your continuation parameter. This will all happen on the coarse mesh.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one
> solve on the coarse level and then interpolate to the next level etc.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before
> they begin their experiments is infinitely more interesting than any
> results to which their experiments lead.
> >>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> >>>>>>> -- Norbert Wiener
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> >>>> -- Norbert Wiener
> >>>
> >>
> >>
> >>
> >>
> >> --
> >> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> >> -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150909/efefb17f/attachment-0001.html>